Jul 28 01:26:08.513756 ip-10-0-128-168 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 01:26:08.940639 ip-10-0-128-168 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:26:08.940639 ip-10-0-128-168 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 01:26:08.940639 ip-10-0-128-168 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:26:08.940639 ip-10-0-128-168 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 01:26:08.940639 ip-10-0-128-168 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:26:08.943673 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.943572    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 01:26:08.949621 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949598    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:26:08.949621 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949617    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:26:08.949621 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949621    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:26:08.949621 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949624    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:26:08.949621 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949627    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:26:08.949621 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949630    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949633    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949637    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949640    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949642    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949645    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949648    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949650    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949653    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949656    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949660    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949664    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949667    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949670    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949674    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949678    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949680    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949683    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949685    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:26:08.949833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949688    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949690    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949693    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949695    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949697    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949700    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949711    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949713    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949716    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949719    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949721    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949724    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949726    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949729    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949732    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949735    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949739    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949741    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949744    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949746    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:26:08.950316 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949749    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949752    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949754    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949757    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949759    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949761    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949764    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949767    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949769    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949772    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949774    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949777    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949779    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949782    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949785    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949787    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949790    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949793    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949795    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949797    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:26:08.950830 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949800    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949802    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949806    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949810    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949812    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949815    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949819    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949823    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949826    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949828    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949831    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949833    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949836    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949839    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949842    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949844    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949847    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949849    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949852    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949855    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:26:08.951318 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949857    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949860    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949862    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949865    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949868    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949870    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949873    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949875    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949878    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949881    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949883    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949886    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949888    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949891    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.949893    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950672    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950680    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950683    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950687    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:26:08.951819 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950690    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950693    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950695    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950698    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950700    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950703    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950705    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950708    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950710    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950713    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950715    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950718    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950720    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950723    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950726    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950729    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950732    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950734    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950736    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950739    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:26:08.952280 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950742    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950744    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950747    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950749    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950752    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950754    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950756    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950759    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950761    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950764    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950766    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950769    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950772    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950775    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950777    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950780    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950783    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950785    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950788    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950790    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:26:08.952803 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950792    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950795    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950797    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950800    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950802    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950805    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950808    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950811    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950813    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950815    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950819    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950821    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950823    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950826    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950828    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950831    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950834    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950836    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950840    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:26:08.953288 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950844    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950847    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950849    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950852    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950854    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950858    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950862    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950865    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950869    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950871    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950874    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950877    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950881    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950884    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950886    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950889    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950891    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950893    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950896    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950899    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:26:08.953769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950902    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950905    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950908    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950912    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950915    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950918    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950921    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950923    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950926    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950929    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950932    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950935    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950937    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950940    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950942    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.950945    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951344    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951350    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951353    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:26:08.954275 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951355    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951359    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951361    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951365    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951368    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951372    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951374    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951377    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951379    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951382    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951384    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951387    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951390    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951393    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951395    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951398    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951401    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951404    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951407    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:26:08.954753 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951409    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951412    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951414    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951417    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951419    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951421    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951424    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951426    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951444    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951447    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951449    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951453    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951455    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951458    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951461    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951467    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951469    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951473    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951475    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951478    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:26:08.955268 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951480    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951483    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951485    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951487    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951490    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951492    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951495    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951498    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951500    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951503    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951505    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951508    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951510    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951513    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951515    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951518    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951520    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951523    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951527    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:26:08.955769 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951531    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951534    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951536    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951539    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951542    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951545    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951548    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951550    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951553    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951555    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951558    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951560    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951562    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951565    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951567    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951570    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951572    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951575    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951577    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:26:08.956222 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951581    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951584    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951587    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951590    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951593    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951595    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951600    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951602    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951605    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951607    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951610    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951612    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951615    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951617    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951620    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951622    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951625    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951628    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.951631    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951700    2576 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951713    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 01:26:08.956725 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951720    2576 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951725    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951729    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951733    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951737    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951741    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951744    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951747    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951750    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951754    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951757    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951760    2576 flags.go:64] FLAG: --cgroup-root=""
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951763    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951766    2576 flags.go:64] FLAG: --client-ca-file=""
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951770    2576 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951773    2576 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951777    2576 flags.go:64] FLAG: --cluster-domain=""
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951780    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951783    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951785    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951789    2576 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951792    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951796    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 01:26:08.957240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951798    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951801    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951804    2576 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951807    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951813    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951816    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951819    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951823    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951826    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951829    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951832    2576 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951835    2576 flags.go:64] FLAG: --enable-server="true"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951838    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951842    2576 flags.go:64] FLAG: --event-burst="100"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951845    2576 flags.go:64] FLAG: --event-qps="50"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951848    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951852    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951854    2576 flags.go:64] FLAG: --eviction-hard=""
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951858    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951861    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951867    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951870    2576 flags.go:64] FLAG: --eviction-soft=""
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951872    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951875    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951879    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 01:26:08.957818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951882    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951884    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951887    2576 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951889    2576 flags.go:64] FLAG: --feature-gates=""
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951893    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951896    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951899    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951902    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951905    2576 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951908    2576 flags.go:64] FLAG: --help="false"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951910    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-128-168.ec2.internal"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951914    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951917    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951920    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951923    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951926    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951929    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951932    2576 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951934    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951937    2576 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951939    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951942    2576 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951945    2576 flags.go:64] FLAG: --kube-reserved=""
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951948    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 01:26:08.958456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951950    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951953    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951956    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951960    2576 flags.go:64] FLAG: --lock-file=""
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951962    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951965    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951967    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951973    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951976    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951979    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951981    2576 flags.go:64] FLAG: --logging-format="text"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951984    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951987    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951990    2576 flags.go:64] FLAG: --manifest-url=""
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951993    2576 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951997    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.951999    2576 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952003    2576 flags.go:64] FLAG: --max-pods="110"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952006    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952010    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952013    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952016    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952018    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952021    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952024    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 01:26:08.959056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952031    2576 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952034    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952037    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952040    2576 flags.go:64] FLAG: --pod-cidr=""
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952043    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952048    2576 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952051    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952053    2576 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952056    2576 flags.go:64] FLAG: --port="10250"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952059    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952062    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bdeb8daae839438f"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952066    2576 flags.go:64] FLAG: --qos-reserved=""
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952069    2576 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952072    2576 flags.go:64] FLAG: --register-node="true"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952075    2576 flags.go:64] FLAG: --register-with-taints=""
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952078    2576 flags.go:64] FLAG: --registry-burst="10"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952081    2576 flags.go:64] FLAG: --registry-qps="5"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952084    2576 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952087    2576 flags.go:64] FLAG: --reserved-memory=""
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952090    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952094    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952096    2576 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952099    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952102    2576 flags.go:64] FLAG: --runonce="false"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952105    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 01:26:08.959687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952107    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952111    2576 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952114    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952117    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952120    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952123    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952125    2576 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952128    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952130    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952133    2576 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952136    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952138    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952141    2576 flags.go:64] FLAG: --system-cgroups=""
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952144    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952149    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952152    2576 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952154    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952157    2576 flags.go:64] FLAG: --tls-min-version=""
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952162    2576 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952165    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952168    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952170    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952173    2576 flags.go:64] FLAG: --v="2"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952177    2576 flags.go:64] FLAG: --version="false"
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952181    2576 flags.go:64] FLAG: --vmodule=""
Jul 28 01:26:08.960318 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952185    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.952188    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952306    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952310    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952313    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952316    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952318    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952321    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952325    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952328    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952330    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952333    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952335    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952338    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952340    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952342    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952345    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952347    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952350    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952352    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:26:08.960994 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952354    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952357    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952359    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952362    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952364    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952368    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952371    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952373    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952376    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952378    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952381    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952383    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952389    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952391    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952394    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952396    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952399    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952401    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952403    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952406    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:26:08.961487 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952414    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952416    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952419    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952421    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952424    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952428    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952444    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952447    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952450    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952453    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952456    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952459    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952462    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952465    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952467    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952470    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952473    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952477    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952479    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:26:08.961970 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952482    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952484    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952487    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952489    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952493    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952497    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952499    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952502    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952504    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952507    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952509    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952512    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952514    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952518    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952520    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952523    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952525    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952528    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952530    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952532    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:26:08.962471 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952535    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952537    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952540    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952542    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952545    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952547    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952549    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952552    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952555    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952557    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952562    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952564    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952566    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952569    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952572    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952574    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952577    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952579    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952582    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952585    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:26:08.962963 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952588    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:26:08.963556 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.952592    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:26:08.963556 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.953291    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:26:08.963556 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.960600    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 01:26:08.963556 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.960622    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 01:26:08.963556 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.961541    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 01:26:08.963556 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.961548    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 01:26:08.963556 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961613    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:26:08.963556 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961619    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:26:08.963556 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961623    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:26:08.963556 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961626    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:26:08.963556 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961628    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:26:08.963556 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961631    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:26:08.963556 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961635    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:26:08.963556 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961638    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961641    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961644    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961647    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961649    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961651    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961654    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961656    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961659    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961661    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961664    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961666    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961669    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961671    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961673    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961676    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961679    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961681    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961684    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961686    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:26:08.964025 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961688    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961691    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961700    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961703    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961706    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961708    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961710    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961713    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961716    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961718    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961721    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961723    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961726    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961728    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961730    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961733    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961735    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961737    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961740    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961742    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:26:08.964530 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961744    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961747    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961749    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961754    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961757    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961760    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961763    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961766    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961769    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961771    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961774    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961776    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961779    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961781    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961785    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961793    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961796    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961799    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961801    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:26:08.965024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961804    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961807    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961811    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961814    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961816    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961819    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961822    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961824    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961827    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961829    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961832    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961835    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961837    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961839    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961842    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961844    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961846    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961849    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961851    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:26:08.965500 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961853    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:26:08.965966 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961856    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:26:08.965966 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961858    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:26:08.965966 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961861    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:26:08.965966 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961863    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:26:08.965966 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961866    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:26:08.965966 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961868    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:26:08.965966 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961870    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:26:08.965966 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961874    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:26:08.965966 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961876    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:26:08.965966 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961884    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:26:08.965966 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961887    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:26:08.965966 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961890    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:26:08.965966 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.961892    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:26:08.965966 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.961897    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:26:08.965966 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962050    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962054    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962057    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962059    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962062    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962064    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962067    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962069    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962072    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962074    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962076    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962078    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962081    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962083    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962085    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962088    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962090    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962092    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962095    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962097    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:26:08.966326 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962100    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962102    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962105    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962107    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962110    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962112    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962115    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962117    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962126    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962129    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962131    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962134    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962136    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962139    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962142    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962146    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962148    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962151    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962153    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:26:08.966829 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962156    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962158    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962160    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962162    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962165    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962168    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962172    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962174    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962177    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962180    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962182    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962185    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962188    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962190    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962193    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962196    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962198    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962201    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962203    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962206    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:26:08.967298 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962208    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962211    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962219    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962222    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962225    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962227    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962229    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962232    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962234    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962236    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962239    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962242    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962244    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962246    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962249    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962251    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962254    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962256    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962258    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962261    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:26:08.967804 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962264    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962266    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962269    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962271    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962273    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962276    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962278    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962281    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962283    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962285    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962288    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962291    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962293    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962295    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962298    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962306    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962308    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962311    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962313    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:26:08.968351 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:08.962315    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:26:08.968910 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.962320    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:26:08.968910 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.962484    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 01:26:08.968910 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.965579    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 01:26:08.968910 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.966961    2576 server.go:1019] "Starting client certificate rotation"
Jul 28 01:26:08.968910 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.967060    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 01:26:08.968910 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.967099    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 01:26:08.970042 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.970028    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 01:26:08.971733 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.971711    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 01:26:08.985086 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.985057    2576 log.go:25] "Validated CRI v1 runtime API"
Jul 28 01:26:08.987218 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.987201    2576 log.go:25] "Validated CRI v1 image API"
Jul 28 01:26:08.988082 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.988062    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 01:26:08.990507 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.990486    2576 fs.go:135] Filesystem UUIDs: map[0c0b09d5-efa9-4d3d-b802-f79cad098dda:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cae29c5f-f05b-4aa5-8e7c-2f582c3adde5:/dev/nvme0n1p3]
Jul 28 01:26:08.990575 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.990506    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 01:26:08.995948 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.995820    2576 manager.go:216] Machine: {Timestamp:2026-07-28 01:26:08.994448191 +0000 UTC m=+0.356436691 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098726 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f89abcaae8899c9a8bc6e6dd81939 SystemUUID:ec2f89ab-caae-8899-c9a8-bc6e6dd81939 BootID:4599f4ae-a85c-4887-80b4-72b49acf3ce9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6f:23:77:28:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6f:23:77:28:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:27:2a:31:ad:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 01:26:08.995948 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.995937    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 01:26:08.996077 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.996031    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 01:26:08.997185 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:08.997168    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 01:26:09.000971 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.000941    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 01:26:09.001115 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.000975    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-168.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 01:26:09.001802 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.001791    2576 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 01:26:09.001833 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.001805    2576 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 01:26:09.001833 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.001812    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 01:26:09.002666 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.002654    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 01:26:09.003585 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.003575    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 01:26:09.005921 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.005909    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 01:26:09.006057 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.006047    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 01:26:09.006092 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.006079    2576 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 01:26:09.006119 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.006102    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 01:26:09.006153 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.006130    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 01:26:09.006153 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.006143    2576 kubelet.go:398] "Adding apiserver pod source"
Jul 28 01:26:09.006153 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.006152    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 01:26:09.006570 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.006557    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 01:26:09.006633 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.006586    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 01:26:09.008859 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.008842    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 01:26:09.010939 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.010919    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 01:26:09.011034 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.010961    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 01:26:09.013279 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.013263    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 01:26:09.013361 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.013282    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 01:26:09.013361 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.013292    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 01:26:09.013361 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.013299    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 01:26:09.013361 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.013311    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 01:26:09.013361 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.013319    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 01:26:09.013361 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.013330    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 01:26:09.013361 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.013338    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 01:26:09.013361 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.013349    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 01:26:09.013361 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.013357    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 01:26:09.013639 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.013376    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 01:26:09.013639 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.013385    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 01:26:09.014267 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.014256    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 01:26:09.014322 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.014270    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 01:26:09.017248 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.017219    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 01:26:09.017332 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.017308    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-168.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 01:26:09.017373 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.017360    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-168.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 01:26:09.017816 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.017803    2576 server.go:1268] "Started kubelet"
Jul 28 01:26:09.017917 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.017883    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 01:26:09.017975 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.017903    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 01:26:09.018028 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.017974    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 01:26:09.022696 ip-10-0-128-168 systemd[1]: Started Kubernetes Kubelet.
Jul 28 01:26:09.024026 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.023808    2576 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 01:26:09.025712 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.025677    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jfrxt"
Jul 28 01:26:09.026748 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.026729    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 01:26:09.029346 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.029327    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 01:26:09.029551 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.028422    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-168.ec2.internal.18c64fbb7fa10564  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-168.ec2.internal,UID:ip-10-0-128-168.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-168.ec2.internal,},FirstTimestamp:2026-07-28 01:26:09.017718116 +0000 UTC m=+0.379706617,LastTimestamp:2026-07-28 01:26:09.017718116 +0000 UTC m=+0.379706617,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-168.ec2.internal,}"
Jul 28 01:26:09.033622 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.033596    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 01:26:09.033702 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.033634    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 01:26:09.033771 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.033717    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Jul 28 01:26:09.034403 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.034244    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 01:26:09.034403 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.034249    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 01:26:09.034403 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.034285    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 01:26:09.034403 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.034351    2576 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 01:26:09.034403 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.034358    2576 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 01:26:09.035148 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.035122    2576 factory.go:55] Registering systemd factory
Jul 28 01:26:09.035214 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.035168    2576 factory.go:223] Registration of the systemd container factory successfully
Jul 28 01:26:09.035380 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.035363    2576 factory.go:153] Registering CRI-O factory
Jul 28 01:26:09.035380 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.035379    2576 factory.go:223] Registration of the crio container factory successfully
Jul 28 01:26:09.035573 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.035468    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 01:26:09.035573 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.035501    2576 factory.go:103] Registering Raw factory
Jul 28 01:26:09.035573 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.035516    2576 manager.go:1196] Started watching for new ooms in manager
Jul 28 01:26:09.035945 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.035929    2576 manager.go:349] Starting recovery of all containers
Jul 28 01:26:09.036562 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.036538    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 01:26:09.036647 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.036598    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-168.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 01:26:09.037720 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.037694    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jfrxt"
Jul 28 01:26:09.047230 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.047084    2576 manager.go:354] Recovery completed
Jul 28 01:26:09.051671 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.051658    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:26:09.053484 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.053466    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:26:09.053555 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.053497    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:26:09.053555 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.053508    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:26:09.054079 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.054065    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 01:26:09.054142 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.054080    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 01:26:09.054142 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.054093    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:26:09.054232 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.054097    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 01:26:09.055111 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.055095    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:26:09.055183 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.055123    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:26:09.055183 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.055144    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:26:09.056048 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.056032    2576 policy_none.go:49] "None policy: Start"
Jul 28 01:26:09.056122 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.056056    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 01:26:09.056122 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.056072    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 01:26:09.056408 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.055962    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-168.ec2.internal.18c64fbb81c2c01a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-168.ec2.internal,UID:ip-10-0-128-168.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-168.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-168.ec2.internal,},FirstTimestamp:2026-07-28 01:26:09.053483034 +0000 UTC m=+0.415471538,LastTimestamp:2026-07-28 01:26:09.053483034 +0000 UTC m=+0.415471538,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-168.ec2.internal,}"
Jul 28 01:26:09.058968 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.058939    2576 policy_none.go:47] "Start"
Jul 28 01:26:09.092916 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.092902    2576 manager.go:337] "Starting Device Plugin manager"
Jul 28 01:26:09.092988 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.092936    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 01:26:09.092988 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.092947    2576 server.go:87] "Starting device plugin registration server"
Jul 28 01:26:09.093223 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.093208    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 01:26:09.093263 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.093226    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 01:26:09.093335 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.093319    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 01:26:09.093467 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.093428    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 01:26:09.093467 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.093459    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 01:26:09.093923 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.093906    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 01:26:09.093998 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.093957    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-168.ec2.internal\" not found"
Jul 28 01:26:09.193761 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.193698    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:26:09.194887 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.194781    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:26:09.194887 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.194811    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:26:09.194887 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.194821    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:26:09.194887 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.194846    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.202208 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.202188    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.202276 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.202218    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-168.ec2.internal\": node \"ip-10-0-128-168.ec2.internal\" not found"
Jul 28 01:26:09.220060 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.220027    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Jul 28 01:26:09.232846 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.232812    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 01:26:09.234383 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.234361    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 01:26:09.234500 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.234412    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 01:26:09.234500 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.234452    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 01:26:09.234500 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.234492    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 01:26:09.236642 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.236621    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:26:09.320898 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.320858    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Jul 28 01:26:09.335091 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.335053    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"]
Jul 28 01:26:09.335246 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.335151    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:26:09.336141 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.336125    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:26:09.336252 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.336198    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:26:09.336252 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.336215    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:26:09.337579 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.337562    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:26:09.337695 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.337678    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.337747 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.337710    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:26:09.338358 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.338341    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:26:09.338460 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.338370    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:26:09.338460 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.338384    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:26:09.338460 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.338387    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:26:09.338460 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.338410    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:26:09.338460 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.338424    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:26:09.339812 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.339798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.339861 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.339826    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:26:09.341638 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.341623    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:26:09.341737 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.341651    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:26:09.341737 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.341664    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:26:09.360314 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.360290    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-168.ec2.internal\" not found" node="ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.364926 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.364908    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-168.ec2.internal\" not found" node="ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.421574 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.421543    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Jul 28 01:26:09.435861 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.435836    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/088df6a04689e2bf1edce0757bca5189-config\") pod \"kube-apiserver-proxy-ip-10-0-128-168.ec2.internal\" (UID: \"088df6a04689e2bf1edce0757bca5189\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.435931 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.435867    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2e5a9fa29df8176300d2ac7d8254fea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal\" (UID: \"a2e5a9fa29df8176300d2ac7d8254fea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.435931 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.435889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2e5a9fa29df8176300d2ac7d8254fea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal\" (UID: \"a2e5a9fa29df8176300d2ac7d8254fea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.522480 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.522395    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Jul 28 01:26:09.536742 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.536722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/088df6a04689e2bf1edce0757bca5189-config\") pod \"kube-apiserver-proxy-ip-10-0-128-168.ec2.internal\" (UID: \"088df6a04689e2bf1edce0757bca5189\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.536794 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.536783    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/088df6a04689e2bf1edce0757bca5189-config\") pod \"kube-apiserver-proxy-ip-10-0-128-168.ec2.internal\" (UID: \"088df6a04689e2bf1edce0757bca5189\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.536833 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.536814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2e5a9fa29df8176300d2ac7d8254fea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal\" (UID: \"a2e5a9fa29df8176300d2ac7d8254fea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.536873 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.536842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2e5a9fa29df8176300d2ac7d8254fea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal\" (UID: \"a2e5a9fa29df8176300d2ac7d8254fea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.536927 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.536888    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2e5a9fa29df8176300d2ac7d8254fea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal\" (UID: \"a2e5a9fa29df8176300d2ac7d8254fea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.536974 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.536916    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2e5a9fa29df8176300d2ac7d8254fea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal\" (UID: \"a2e5a9fa29df8176300d2ac7d8254fea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.622944 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.622911    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Jul 28 01:26:09.664511 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.664479    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.668043 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:09.668020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Jul 28 01:26:09.723949 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.723912    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Jul 28 01:26:09.824557 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:09.824487    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Jul 28 01:26:11.115960 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:11.115921    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Jul 28 01:26:11.157512 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:11.157479    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 01:26:11.157983 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:11.157661    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 01:26:11.216729 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:11.216699    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Jul 28 01:26:11.234582 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:11.234546    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 01:21:10 +0000 UTC" deadline="2028-04-19 12:26:12.197312666 +0000 UTC"
Jul 28 01:26:11.234582 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:11.234578    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15155h0m0.962738328s"
Jul 28 01:26:11.316851 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:11.316798    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Jul 28 01:26:11.406817 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:11.406787    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:26:11.417740 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:11.417720    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-168.ec2.internal\" not found"
Jul 28 01:26:11.422886 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:11.422843    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088df6a04689e2bf1edce0757bca5189.slice/crio-c205d77be89ef2a4b8934686db8dacadf96cad638a192866df041f0a7643c363 WatchSource:0}: Error finding container c205d77be89ef2a4b8934686db8dacadf96cad638a192866df041f0a7643c363: Status 404 returned error can't find the container with id c205d77be89ef2a4b8934686db8dacadf96cad638a192866df041f0a7643c363
Jul 28 01:26:11.423428 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:11.423403    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e5a9fa29df8176300d2ac7d8254fea.slice/crio-d847384960c0f68a95b3c0f9782a7f547bd0dc5d8af72828f342a0abab3cc6ac WatchSource:0}: Error finding container d847384960c0f68a95b3c0f9782a7f547bd0dc5d8af72828f342a0abab3cc6ac: Status 404 returned error can't find the container with id d847384960c0f68a95b3c0f9782a7f547bd0dc5d8af72828f342a0abab3cc6ac
Jul 28 01:26:11.429492 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:11.429220    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:26:11.431226 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:11.431161    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal" event={"ID":"088df6a04689e2bf1edce0757bca5189","Type":"ContainerStarted","Data":"c205d77be89ef2a4b8934686db8dacadf96cad638a192866df041f0a7643c363"}
Jul 28 01:26:11.432545 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:11.432526    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal" event={"ID":"a2e5a9fa29df8176300d2ac7d8254fea","Type":"ContainerStarted","Data":"d847384960c0f68a95b3c0f9782a7f547bd0dc5d8af72828f342a0abab3cc6ac"}
Jul 28 01:26:11.490090 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:11.490060    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:26:11.525758 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:11.525733    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal"
Jul 28 01:26:11.530536 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:11.530521    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:26:11.539736 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:11.539718    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 01:26:11.540667 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:11.540654    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal"
Jul 28 01:26:11.563221 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:11.563197    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 01:26:12.198478 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.198449    2576 apiserver.go:52] "Watching apiserver"
Jul 28 01:26:12.205397 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.205359    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 01:26:12.205741 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.205721    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-kwgbx","openshift-network-diagnostics/network-check-target-km9pg","openshift-ovn-kubernetes/ovnkube-node-v8dgm","kube-system/global-pull-secret-syncer-nf7cb","kube-system/konnectivity-agent-cdqxp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal","openshift-multus/multus-2h5rf","openshift-multus/network-metrics-daemon-22fhh","openshift-network-operator/iptables-alerter-p4bgp","kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal","openshift-cluster-node-tuning-operator/tuned-jh4fs","openshift-dns/node-resolver-jbm5x","openshift-image-registry/node-ca-q2vqt"]
Jul 28 01:26:12.207907 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.207886    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p4bgp"
Jul 28 01:26:12.208919 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.208899    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jbm5x"
Jul 28 01:26:12.210472 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.210336    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:26:12.210472 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.210355    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 01:26:12.210472 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.210367    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5t7gd\""
Jul 28 01:26:12.210472 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.210427    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.210704 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.210455    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 01:26:12.211134 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.211116    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gmbq2\""
Jul 28 01:26:12.211233 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.211182    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 01:26:12.211428 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.211413    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 01:26:12.214027 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.211706    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:12.214027 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.211780    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:12.214027 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.212709    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 01:26:12.214027 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.212761    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 01:26:12.214027 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.213082    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cdqxp"
Jul 28 01:26:12.214027 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.213087    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qgtl7\""
Jul 28 01:26:12.214027 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.213354    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 01:26:12.214399 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.214335    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.215172 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.214953    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 01:26:12.215172 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.215047    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 01:26:12.215172 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.215091    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 01:26:12.216643 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.216350    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:12.216643 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.216461    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:12.219540 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.218821    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q2vqt"
Jul 28 01:26:12.220326 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.220308    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:12.220429 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.220375    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:12.222783 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.222755    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.222966 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.222948    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 01:26:12.223294 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.223273    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 01:26:12.223372 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.223341    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v85qb\""
Jul 28 01:26:12.223453 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.223440    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 01:26:12.223548 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.223528    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 01:26:12.223602 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.223551    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gw7ts\""
Jul 28 01:26:12.223654 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.223643    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 01:26:12.223735 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.223717    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 01:26:12.223815 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.223792    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gqnqs\""
Jul 28 01:26:12.223921 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.223814    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 01:26:12.223974 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.223927    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 01:26:12.226658 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.226640    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qg8n9\""
Jul 28 01:26:12.226886 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.226869    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 01:26:12.227206 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.227185    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 01:26:12.227342 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.227330    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.227713 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.227494    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.228685 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.228666    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 01:26:12.231754 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.231731    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 01:26:12.231843 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.231770    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 01:26:12.231931 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.231912    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 01:26:12.232027 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.232000    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 01:26:12.232224 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.232212    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:26:12.232333 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.232311    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 01:26:12.232602 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.232586    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rvgf2\""
Jul 28 01:26:12.233181 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.233158    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 01:26:12.233310 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.233263    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zbmt2\""
Jul 28 01:26:12.237454 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237434    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc143eaf-873e-484b-9ef0-adc12e645c43-tmp-dir\") pod \"node-resolver-jbm5x\" (UID: \"bc143eaf-873e-484b-9ef0-adc12e645c43\") " pod="openshift-dns/node-resolver-jbm5x"
Jul 28 01:26:12.237571 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237470    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-run-openvswitch\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.237571 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237488    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dlhj\" (UniqueName: \"kubernetes.io/projected/61c607d9-7b10-4b44-88ee-e81ada7ef179-kube-api-access-5dlhj\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.237571 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdmsj\" (UniqueName: \"kubernetes.io/projected/d356f315-756f-439b-882e-35320cae4a10-kube-api-access-rdmsj\") pod \"iptables-alerter-p4bgp\" (UID: \"d356f315-756f-439b-882e-35320cae4a10\") " pod="openshift-network-operator/iptables-alerter-p4bgp"
Jul 28 01:26:12.237571 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237523    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-socket-dir\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.237571 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237543    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-systemd-units\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.237571 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-node-log\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.237838 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237592    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbq59\" (UniqueName: \"kubernetes.io/projected/4db16d2d-9789-4546-96e8-3438dd7fdba8-kube-api-access-mbq59\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.237838 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237618    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-multus-cni-dir\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.237838 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-cnibin\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.237838 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08fb009b-b1f6-465f-b258-7e408d59d0d4-tmp\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.237838 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be103d49-24c3-4a2e-9a55-2fcc960c1a71-kubelet-config\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:12.237838 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237763    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-var-lib-kubelet\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.237838 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237791    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-var-lib-kubelet\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.237838 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237813    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-kubelet\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.238205 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.238205 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237885    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-etc-selinux\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.238205 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-sys-fs\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.238205 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89eecf33-ed8e-41fe-8fce-48a72361f9ad-cni-binary-copy\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.238205 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.237991    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-run-multus-certs\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.238205 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238024    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-sysconfig\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.238205 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238134    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-run-ovn\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.238205 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238175    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d356f315-756f-439b-882e-35320cae4a10-host-slash\") pod \"iptables-alerter-p4bgp\" (UID: \"d356f315-756f-439b-882e-35320cae4a10\") " pod="openshift-network-operator/iptables-alerter-p4bgp"
Jul 28 01:26:12.238641 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-os-release\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.238641 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-multus-socket-dir-parent\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.238641 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-multus-conf-dir\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.238641 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh75q\" (UniqueName: \"kubernetes.io/projected/89eecf33-ed8e-41fe-8fce-48a72361f9ad-kube-api-access-sh75q\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.238641 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqkct\" (UniqueName: \"kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct\") pod \"network-check-target-km9pg\" (UID: \"14745d0d-a327-4289-84d7-213c1e2f3eef\") " pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:12.238641 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238369    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-system-cni-dir\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.238641 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x84qv\" (UniqueName: \"kubernetes.io/projected/bc143eaf-873e-484b-9ef0-adc12e645c43-kube-api-access-x84qv\") pod \"node-resolver-jbm5x\" (UID: \"bc143eaf-873e-484b-9ef0-adc12e645c43\") " pod="openshift-dns/node-resolver-jbm5x"
Jul 28 01:26:12.238641 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238454    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-run-systemd\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.238641 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238478    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4db16d2d-9789-4546-96e8-3438dd7fdba8-cnibin\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.238641 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4db16d2d-9789-4546-96e8-3438dd7fdba8-os-release\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.238641 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4db16d2d-9789-4546-96e8-3438dd7fdba8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.238641 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4db16d2d-9789-4546-96e8-3438dd7fdba8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.238641 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-run-ovn-kubernetes\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61c607d9-7b10-4b44-88ee-e81ada7ef179-ovnkube-config\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tnlv\" (UniqueName: \"kubernetes.io/projected/45a5d7f9-5b74-4f84-a448-d89cb7445b23-kube-api-access-5tnlv\") pod \"node-ca-q2vqt\" (UID: \"45a5d7f9-5b74-4f84-a448-d89cb7445b23\") " pod="openshift-image-registry/node-ca-q2vqt"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238716    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-var-lib-cni-bin\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-var-lib-cni-multus\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bc143eaf-873e-484b-9ef0-adc12e645c43-hosts-file\") pod \"node-resolver-jbm5x\" (UID: \"bc143eaf-873e-484b-9ef0-adc12e645c43\") " pod="openshift-dns/node-resolver-jbm5x"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238789    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4db16d2d-9789-4546-96e8-3438dd7fdba8-cni-binary-copy\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4db16d2d-9789-4546-96e8-3438dd7fdba8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238848    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-kubernetes\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-sys\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-lib-modules\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.238942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-device-dir\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-run\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61c607d9-7b10-4b44-88ee-e81ada7ef179-ovn-node-metrics-cert\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61c607d9-7b10-4b44-88ee-e81ada7ef179-ovnkube-script-lib\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.239401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239194    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-log-socket\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-cni-netd\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4db16d2d-9789-4546-96e8-3438dd7fdba8-system-cni-dir\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-host\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239302    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-slash\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45a5d7f9-5b74-4f84-a448-d89cb7445b23-host\") pod \"node-ca-q2vqt\" (UID: \"45a5d7f9-5b74-4f84-a448-d89cb7445b23\") " pod="openshift-image-registry/node-ca-q2vqt"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239369    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45a5d7f9-5b74-4f84-a448-d89cb7445b23-serviceca\") pod \"node-ca-q2vqt\" (UID: \"45a5d7f9-5b74-4f84-a448-d89cb7445b23\") " pod="openshift-image-registry/node-ca-q2vqt"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-run-k8s-cni-cncf-io\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-systemd\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239516    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3aa211a5-d538-4fd8-a415-29ea6de08067-agent-certs\") pod \"konnectivity-agent-cdqxp\" (UID: \"3aa211a5-d538-4fd8-a415-29ea6de08067\") " pod="kube-system/konnectivity-agent-cdqxp"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239565    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3aa211a5-d538-4fd8-a415-29ea6de08067-konnectivity-ca\") pod \"konnectivity-agent-cdqxp\" (UID: \"3aa211a5-d538-4fd8-a415-29ea6de08067\") " pod="kube-system/konnectivity-agent-cdqxp"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89eecf33-ed8e-41fe-8fce-48a72361f9ad-multus-daemon-config\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239612    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-var-lib-openvswitch\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239635    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-529hr\" (UniqueName: \"kubernetes.io/projected/49687516-e291-414c-95b7-9f319406a20d-kube-api-access-529hr\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-modprobe-d\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.240184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239694    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-tuned\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.240923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239716    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6tpd\" (UniqueName: \"kubernetes.io/projected/08fb009b-b1f6-465f-b258-7e408d59d0d4-kube-api-access-z6tpd\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.240923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239740    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-run-netns\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.240923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239763    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-cni-bin\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.240923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-hostroot\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.240923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239803    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61c607d9-7b10-4b44-88ee-e81ada7ef179-env-overrides\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.240923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lpm7\" (UniqueName: \"kubernetes.io/projected/8849435f-9f9d-4fd2-858c-b037077d839c-kube-api-access-4lpm7\") pod \"network-metrics-daemon-22fhh\" (UID: \"8849435f-9f9d-4fd2-858c-b037077d839c\") " pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:12.240923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-registration-dir\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.240923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-run-netns\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.240923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-etc-kubernetes\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.240923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-sysctl-d\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.240923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-sysctl-conf\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.240923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-etc-openvswitch\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.240923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.239998    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be103d49-24c3-4a2e-9a55-2fcc960c1a71-dbus\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:12.240923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.240019    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs\") pod \"network-metrics-daemon-22fhh\" (UID: \"8849435f-9f9d-4fd2-858c-b037077d839c\") " pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:12.240923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.240042    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d356f315-756f-439b-882e-35320cae4a10-iptables-alerter-script\") pod \"iptables-alerter-p4bgp\" (UID: \"d356f315-756f-439b-882e-35320cae4a10\") " pod="openshift-network-operator/iptables-alerter-p4bgp"
Jul 28 01:26:12.247416 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.247396    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 01:26:12.318708 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.318675    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t4668"
Jul 28 01:26:12.327716 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.327691    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t4668"
Jul 28 01:26:12.341211 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341182    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-kubernetes\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.341368 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341216    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-sys\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.341368 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-lib-modules\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.341368 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341263    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:12.341368 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-device-dir\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.341368 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-run\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.341368 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341316    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-sys\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.341368 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61c607d9-7b10-4b44-88ee-e81ada7ef179-ovn-node-metrics-cert\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.341368 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341316    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-kubernetes\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341410    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-lib-modules\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341419    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-device-dir\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.341428    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341458    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-run\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.341496    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret podName:be103d49-24c3-4a2e-9a55-2fcc960c1a71 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:12.841478573 +0000 UTC m=+3.012517592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret") pod "global-pull-secret-syncer-nf7cb" (UID: "be103d49-24c3-4a2e-9a55-2fcc960c1a71") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341520    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61c607d9-7b10-4b44-88ee-e81ada7ef179-ovnkube-script-lib\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341549    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-log-socket\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341570    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-cni-netd\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4db16d2d-9789-4546-96e8-3438dd7fdba8-system-cni-dir\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341641    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4db16d2d-9789-4546-96e8-3438dd7fdba8-system-cni-dir\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341643    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-log-socket\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341662    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-host\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341682    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-cni-netd\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341737    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-host\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-slash\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45a5d7f9-5b74-4f84-a448-d89cb7445b23-host\") pod \"node-ca-q2vqt\" (UID: \"45a5d7f9-5b74-4f84-a448-d89cb7445b23\") " pod="openshift-image-registry/node-ca-q2vqt"
Jul 28 01:26:12.341856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45a5d7f9-5b74-4f84-a448-d89cb7445b23-serviceca\") pod \"node-ca-q2vqt\" (UID: \"45a5d7f9-5b74-4f84-a448-d89cb7445b23\") " pod="openshift-image-registry/node-ca-q2vqt"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341844    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-slash\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-run-k8s-cni-cncf-io\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341882    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45a5d7f9-5b74-4f84-a448-d89cb7445b23-host\") pod \"node-ca-q2vqt\" (UID: \"45a5d7f9-5b74-4f84-a448-d89cb7445b23\") " pod="openshift-image-registry/node-ca-q2vqt"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341893    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-systemd\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341919    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3aa211a5-d538-4fd8-a415-29ea6de08067-agent-certs\") pod \"konnectivity-agent-cdqxp\" (UID: \"3aa211a5-d538-4fd8-a415-29ea6de08067\") " pod="kube-system/konnectivity-agent-cdqxp"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3aa211a5-d538-4fd8-a415-29ea6de08067-konnectivity-ca\") pod \"konnectivity-agent-cdqxp\" (UID: \"3aa211a5-d538-4fd8-a415-29ea6de08067\") " pod="kube-system/konnectivity-agent-cdqxp"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.341999    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89eecf33-ed8e-41fe-8fce-48a72361f9ad-multus-daemon-config\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342034    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-run-k8s-cni-cncf-io\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342047    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-systemd\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342050    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-var-lib-openvswitch\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342197    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-var-lib-openvswitch\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342196    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-529hr\" (UniqueName: \"kubernetes.io/projected/49687516-e291-414c-95b7-9f319406a20d-kube-api-access-529hr\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-modprobe-d\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-tuned\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.342726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6tpd\" (UniqueName: \"kubernetes.io/projected/08fb009b-b1f6-465f-b258-7e408d59d0d4-kube-api-access-z6tpd\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342314    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-run-netns\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342317    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61c607d9-7b10-4b44-88ee-e81ada7ef179-ovnkube-script-lib\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342336    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-cni-bin\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342339    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45a5d7f9-5b74-4f84-a448-d89cb7445b23-serviceca\") pod \"node-ca-q2vqt\" (UID: \"45a5d7f9-5b74-4f84-a448-d89cb7445b23\") " pod="openshift-image-registry/node-ca-q2vqt"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342360    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-hostroot\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61c607d9-7b10-4b44-88ee-e81ada7ef179-env-overrides\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342473    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lpm7\" (UniqueName: \"kubernetes.io/projected/8849435f-9f9d-4fd2-858c-b037077d839c-kube-api-access-4lpm7\") pod \"network-metrics-daemon-22fhh\" (UID: \"8849435f-9f9d-4fd2-858c-b037077d839c\") " pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-registration-dir\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342506    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-modprobe-d\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342527    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-run-netns\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342532    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-cni-bin\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342558    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89eecf33-ed8e-41fe-8fce-48a72361f9ad-multus-daemon-config\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342578    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-hostroot\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342578    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-run-netns\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-etc-kubernetes\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342617    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-run-netns\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342675    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-sysctl-d\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.343437 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342687    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-etc-kubernetes\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342698    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3aa211a5-d538-4fd8-a415-29ea6de08067-konnectivity-ca\") pod \"konnectivity-agent-cdqxp\" (UID: \"3aa211a5-d538-4fd8-a415-29ea6de08067\") " pod="kube-system/konnectivity-agent-cdqxp"
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342734    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-sysctl-conf\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342763    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-registration-dir\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342800    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-sysctl-d\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342804    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-etc-openvswitch\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be103d49-24c3-4a2e-9a55-2fcc960c1a71-dbus\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs\") pod \"network-metrics-daemon-22fhh\" (UID: \"8849435f-9f9d-4fd2-858c-b037077d839c\") " pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342884    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61c607d9-7b10-4b44-88ee-e81ada7ef179-env-overrides\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d356f315-756f-439b-882e-35320cae4a10-iptables-alerter-script\") pod \"iptables-alerter-p4bgp\" (UID: \"d356f315-756f-439b-882e-35320cae4a10\") " pod="openshift-network-operator/iptables-alerter-p4bgp"
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342931    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc143eaf-873e-484b-9ef0-adc12e645c43-tmp-dir\") pod \"node-resolver-jbm5x\" (UID: \"bc143eaf-873e-484b-9ef0-adc12e645c43\") " pod="openshift-dns/node-resolver-jbm5x"
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342939    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-etc-openvswitch\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342947    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-sysctl-conf\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.342963    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.342975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-run-openvswitch\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.343021    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs podName:8849435f-9f9d-4fd2-858c-b037077d839c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:12.843005914 +0000 UTC m=+3.014044950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs") pod "network-metrics-daemon-22fhh" (UID: "8849435f-9f9d-4fd2-858c-b037077d839c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dlhj\" (UniqueName: \"kubernetes.io/projected/61c607d9-7b10-4b44-88ee-e81ada7ef179-kube-api-access-5dlhj\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.344248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343142    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be103d49-24c3-4a2e-9a55-2fcc960c1a71-dbus\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343272    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-run-openvswitch\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343248    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdmsj\" (UniqueName: \"kubernetes.io/projected/d356f315-756f-439b-882e-35320cae4a10-kube-api-access-rdmsj\") pod \"iptables-alerter-p4bgp\" (UID: \"d356f315-756f-439b-882e-35320cae4a10\") " pod="openshift-network-operator/iptables-alerter-p4bgp"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343293    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc143eaf-873e-484b-9ef0-adc12e645c43-tmp-dir\") pod \"node-resolver-jbm5x\" (UID: \"bc143eaf-873e-484b-9ef0-adc12e645c43\") " pod="openshift-dns/node-resolver-jbm5x"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343304    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-socket-dir\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-systemd-units\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-node-log\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343373    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbq59\" (UniqueName: \"kubernetes.io/projected/4db16d2d-9789-4546-96e8-3438dd7fdba8-kube-api-access-mbq59\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-multus-cni-dir\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343437    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d356f315-756f-439b-882e-35320cae4a10-iptables-alerter-script\") pod \"iptables-alerter-p4bgp\" (UID: \"d356f315-756f-439b-882e-35320cae4a10\") " pod="openshift-network-operator/iptables-alerter-p4bgp"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343448    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-socket-dir\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343480    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-systemd-units\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343497    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-cnibin\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-cnibin\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343517    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-node-log\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343536    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08fb009b-b1f6-465f-b258-7e408d59d0d4-tmp\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343560    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-multus-cni-dir\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be103d49-24c3-4a2e-9a55-2fcc960c1a71-kubelet-config\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:12.345039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-var-lib-kubelet\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-var-lib-kubelet\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343653    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-var-lib-kubelet\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343652    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be103d49-24c3-4a2e-9a55-2fcc960c1a71-kubelet-config\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-kubelet\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343698    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-var-lib-kubelet\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343719    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343746    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-kubelet\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-etc-selinux\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-sys-fs\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343803    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343816    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89eecf33-ed8e-41fe-8fce-48a72361f9ad-cni-binary-copy\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343838    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-run-multus-certs\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-sysconfig\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343877    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-sys-fs\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343884    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49687516-e291-414c-95b7-9f319406a20d-etc-selinux\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-run-ovn\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.345750 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d356f315-756f-439b-882e-35320cae4a10-host-slash\") pod \"iptables-alerter-p4bgp\" (UID: \"d356f315-756f-439b-882e-35320cae4a10\") " pod="openshift-network-operator/iptables-alerter-p4bgp"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343932    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-run-ovn\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343936    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-sysconfig\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-os-release\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343965    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-run-multus-certs\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.343999    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-multus-socket-dir-parent\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344024    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-multus-conf-dir\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344047    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d356f315-756f-439b-882e-35320cae4a10-host-slash\") pod \"iptables-alerter-p4bgp\" (UID: \"d356f315-756f-439b-882e-35320cae4a10\") " pod="openshift-network-operator/iptables-alerter-p4bgp"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344097    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-multus-socket-dir-parent\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344130    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-multus-conf-dir\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344134    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh75q\" (UniqueName: \"kubernetes.io/projected/89eecf33-ed8e-41fe-8fce-48a72361f9ad-kube-api-access-sh75q\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344146    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-os-release\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344176    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqkct\" (UniqueName: \"kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct\") pod \"network-check-target-km9pg\" (UID: \"14745d0d-a327-4289-84d7-213c1e2f3eef\") " pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344265    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89eecf33-ed8e-41fe-8fce-48a72361f9ad-cni-binary-copy\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-system-cni-dir\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344428    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-system-cni-dir\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x84qv\" (UniqueName: \"kubernetes.io/projected/bc143eaf-873e-484b-9ef0-adc12e645c43-kube-api-access-x84qv\") pod \"node-resolver-jbm5x\" (UID: \"bc143eaf-873e-484b-9ef0-adc12e645c43\") " pod="openshift-dns/node-resolver-jbm5x"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344468    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-run-systemd\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.346324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344494    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4db16d2d-9789-4546-96e8-3438dd7fdba8-cnibin\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344533    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-run-systemd\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344537    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4db16d2d-9789-4546-96e8-3438dd7fdba8-cnibin\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4db16d2d-9789-4546-96e8-3438dd7fdba8-os-release\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4db16d2d-9789-4546-96e8-3438dd7fdba8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344620    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4db16d2d-9789-4546-96e8-3438dd7fdba8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-run-ovn-kubernetes\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344696    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4db16d2d-9789-4546-96e8-3438dd7fdba8-os-release\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61c607d9-7b10-4b44-88ee-e81ada7ef179-ovnkube-config\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344742    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61c607d9-7b10-4b44-88ee-e81ada7ef179-host-run-ovn-kubernetes\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tnlv\" (UniqueName: \"kubernetes.io/projected/45a5d7f9-5b74-4f84-a448-d89cb7445b23-kube-api-access-5tnlv\") pod \"node-ca-q2vqt\" (UID: \"45a5d7f9-5b74-4f84-a448-d89cb7445b23\") " pod="openshift-image-registry/node-ca-q2vqt"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-var-lib-cni-bin\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-var-lib-cni-multus\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bc143eaf-873e-484b-9ef0-adc12e645c43-hosts-file\") pod \"node-resolver-jbm5x\" (UID: \"bc143eaf-873e-484b-9ef0-adc12e645c43\") " pod="openshift-dns/node-resolver-jbm5x"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4db16d2d-9789-4546-96e8-3438dd7fdba8-cni-binary-copy\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.344892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4db16d2d-9789-4546-96e8-3438dd7fdba8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.345062    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4db16d2d-9789-4546-96e8-3438dd7fdba8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.346879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.345159    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61c607d9-7b10-4b44-88ee-e81ada7ef179-ovnkube-config\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.347476 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.345218    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-var-lib-cni-multus\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.347476 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.345248    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4db16d2d-9789-4546-96e8-3438dd7fdba8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.347476 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.345260    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89eecf33-ed8e-41fe-8fce-48a72361f9ad-host-var-lib-cni-bin\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.347476 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.345296    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bc143eaf-873e-484b-9ef0-adc12e645c43-hosts-file\") pod \"node-resolver-jbm5x\" (UID: \"bc143eaf-873e-484b-9ef0-adc12e645c43\") " pod="openshift-dns/node-resolver-jbm5x"
Jul 28 01:26:12.347476 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.345688    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4db16d2d-9789-4546-96e8-3438dd7fdba8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.347476 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.345793    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4db16d2d-9789-4546-96e8-3438dd7fdba8-cni-binary-copy\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.347476 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.346238    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08fb009b-b1f6-465f-b258-7e408d59d0d4-tmp\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.347476 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.346416    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08fb009b-b1f6-465f-b258-7e408d59d0d4-etc-tuned\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.347476 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.346551    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3aa211a5-d538-4fd8-a415-29ea6de08067-agent-certs\") pod \"konnectivity-agent-cdqxp\" (UID: \"3aa211a5-d538-4fd8-a415-29ea6de08067\") " pod="kube-system/konnectivity-agent-cdqxp"
Jul 28 01:26:12.349016 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.348995    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61c607d9-7b10-4b44-88ee-e81ada7ef179-ovn-node-metrics-cert\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.356016 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.355980    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:12.356016 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.356004    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:12.356016 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.356015    2576 projected.go:196] Error preparing data for projected volume kube-api-access-hqkct for pod openshift-network-diagnostics/network-check-target-km9pg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:12.356228 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.356058    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct podName:14745d0d-a327-4289-84d7-213c1e2f3eef nodeName:}" failed. No retries permitted until 2026-07-28 01:26:12.856042851 +0000 UTC m=+3.027081869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hqkct" (UniqueName: "kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct") pod "network-check-target-km9pg" (UID: "14745d0d-a327-4289-84d7-213c1e2f3eef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:12.358545 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.358475    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh75q\" (UniqueName: \"kubernetes.io/projected/89eecf33-ed8e-41fe-8fce-48a72361f9ad-kube-api-access-sh75q\") pod \"multus-2h5rf\" (UID: \"89eecf33-ed8e-41fe-8fce-48a72361f9ad\") " pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.358818 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.358793    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbq59\" (UniqueName: \"kubernetes.io/projected/4db16d2d-9789-4546-96e8-3438dd7fdba8-kube-api-access-mbq59\") pod \"multus-additional-cni-plugins-kwgbx\" (UID: \"4db16d2d-9789-4546-96e8-3438dd7fdba8\") " pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.358998 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.358950    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6tpd\" (UniqueName: \"kubernetes.io/projected/08fb009b-b1f6-465f-b258-7e408d59d0d4-kube-api-access-z6tpd\") pod \"tuned-jh4fs\" (UID: \"08fb009b-b1f6-465f-b258-7e408d59d0d4\") " pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.360339 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.360313    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdmsj\" (UniqueName: \"kubernetes.io/projected/d356f315-756f-439b-882e-35320cae4a10-kube-api-access-rdmsj\") pod \"iptables-alerter-p4bgp\" (UID: \"d356f315-756f-439b-882e-35320cae4a10\") " pod="openshift-network-operator/iptables-alerter-p4bgp"
Jul 28 01:26:12.360792 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.360766    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-529hr\" (UniqueName: \"kubernetes.io/projected/49687516-e291-414c-95b7-9f319406a20d-kube-api-access-529hr\") pod \"aws-ebs-csi-driver-node-n2gb6\" (UID: \"49687516-e291-414c-95b7-9f319406a20d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.361526 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.361482    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dlhj\" (UniqueName: \"kubernetes.io/projected/61c607d9-7b10-4b44-88ee-e81ada7ef179-kube-api-access-5dlhj\") pod \"ovnkube-node-v8dgm\" (UID: \"61c607d9-7b10-4b44-88ee-e81ada7ef179\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.361958 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.361899    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tnlv\" (UniqueName: \"kubernetes.io/projected/45a5d7f9-5b74-4f84-a448-d89cb7445b23-kube-api-access-5tnlv\") pod \"node-ca-q2vqt\" (UID: \"45a5d7f9-5b74-4f84-a448-d89cb7445b23\") " pod="openshift-image-registry/node-ca-q2vqt"
Jul 28 01:26:12.363027 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.363000    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x84qv\" (UniqueName: \"kubernetes.io/projected/bc143eaf-873e-484b-9ef0-adc12e645c43-kube-api-access-x84qv\") pod \"node-resolver-jbm5x\" (UID: \"bc143eaf-873e-484b-9ef0-adc12e645c43\") " pod="openshift-dns/node-resolver-jbm5x"
Jul 28 01:26:12.363506 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.363488    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lpm7\" (UniqueName: \"kubernetes.io/projected/8849435f-9f9d-4fd2-858c-b037077d839c-kube-api-access-4lpm7\") pod \"network-metrics-daemon-22fhh\" (UID: \"8849435f-9f9d-4fd2-858c-b037077d839c\") " pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:12.521506 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.521421    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p4bgp"
Jul 28 01:26:12.529091 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.529068    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jbm5x"
Jul 28 01:26:12.537561 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.537536    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:12.545173 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.545156    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cdqxp"
Jul 28 01:26:12.553941 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.553917    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6"
Jul 28 01:26:12.560713 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.560680    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q2vqt"
Jul 28 01:26:12.568481 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.568450    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kwgbx"
Jul 28 01:26:12.575255 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.575236    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2h5rf"
Jul 28 01:26:12.581104 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.581086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jh4fs"
Jul 28 01:26:12.620159 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.620124    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:26:12.848794 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.848718    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:12.848794 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.848763    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs\") pod \"network-metrics-daemon-22fhh\" (UID: \"8849435f-9f9d-4fd2-858c-b037077d839c\") " pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:12.848961 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.848873    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:12.848961 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.848883    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:12.848961 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.848934    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs podName:8849435f-9f9d-4fd2-858c-b037077d839c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:13.84892164 +0000 UTC m=+4.019960657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs") pod "network-metrics-daemon-22fhh" (UID: "8849435f-9f9d-4fd2-858c-b037077d839c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:12.848961 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.848946    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret podName:be103d49-24c3-4a2e-9a55-2fcc960c1a71 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:13.848940321 +0000 UTC m=+4.019979337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret") pod "global-pull-secret-syncer-nf7cb" (UID: "be103d49-24c3-4a2e-9a55-2fcc960c1a71") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:12.949434 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:12.949397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqkct\" (UniqueName: \"kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct\") pod \"network-check-target-km9pg\" (UID: \"14745d0d-a327-4289-84d7-213c1e2f3eef\") " pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:12.949614 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.949525    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:12.949614 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.949549    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:12.949614 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.949561    2576 projected.go:196] Error preparing data for projected volume kube-api-access-hqkct for pod openshift-network-diagnostics/network-check-target-km9pg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:12.949774 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:12.949630    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct podName:14745d0d-a327-4289-84d7-213c1e2f3eef nodeName:}" failed. No retries permitted until 2026-07-28 01:26:13.949609291 +0000 UTC m=+4.120648323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqkct" (UniqueName: "kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct") pod "network-check-target-km9pg" (UID: "14745d0d-a327-4289-84d7-213c1e2f3eef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:13.329753 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:13.329715    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:21:12 +0000 UTC" deadline="2027-12-23 19:39:41.907798852 +0000 UTC"
Jul 28 01:26:13.329753 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:13.329751    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12330h13m28.578052722s"
Jul 28 01:26:13.330793 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:13.330765    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49687516_e291_414c_95b7_9f319406a20d.slice/crio-80f44a3bbbc75bf8ceafd806f7201fc2394f876474fa1955693cafcc80c93835 WatchSource:0}: Error finding container 80f44a3bbbc75bf8ceafd806f7201fc2394f876474fa1955693cafcc80c93835: Status 404 returned error can't find the container with id 80f44a3bbbc75bf8ceafd806f7201fc2394f876474fa1955693cafcc80c93835
Jul 28 01:26:13.331585 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:13.331561    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a5d7f9_5b74_4f84_a448_d89cb7445b23.slice/crio-e16741ce414f4a99d7b83253a138e3869af3ad5ed1eba2df40d4045e50cde096 WatchSource:0}: Error finding container e16741ce414f4a99d7b83253a138e3869af3ad5ed1eba2df40d4045e50cde096: Status 404 returned error can't find the container with id e16741ce414f4a99d7b83253a138e3869af3ad5ed1eba2df40d4045e50cde096
Jul 28 01:26:13.332323 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:13.332303    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db16d2d_9789_4546_96e8_3438dd7fdba8.slice/crio-d4cec1ec48c52b3dd08c1991c4c2f31161feb62bb7ffe9fce47ddd413273e9ff WatchSource:0}: Error finding container d4cec1ec48c52b3dd08c1991c4c2f31161feb62bb7ffe9fce47ddd413273e9ff: Status 404 returned error can't find the container with id d4cec1ec48c52b3dd08c1991c4c2f31161feb62bb7ffe9fce47ddd413273e9ff
Jul 28 01:26:13.334015 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:13.333688    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa211a5_d538_4fd8_a415_29ea6de08067.slice/crio-364afef3e8b0f325e4bc04e12d10d85f45c1a23aa7141fc022b5c39df1df5531 WatchSource:0}: Error finding container 364afef3e8b0f325e4bc04e12d10d85f45c1a23aa7141fc022b5c39df1df5531: Status 404 returned error can't find the container with id 364afef3e8b0f325e4bc04e12d10d85f45c1a23aa7141fc022b5c39df1df5531
Jul 28 01:26:13.334991 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:13.334810    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd356f315_756f_439b_882e_35320cae4a10.slice/crio-c5d88f7d84f4d636bdce73c31c420ffb29340ec79a17a863fe0778a86d727549 WatchSource:0}: Error finding container c5d88f7d84f4d636bdce73c31c420ffb29340ec79a17a863fe0778a86d727549: Status 404 returned error can't find the container with id c5d88f7d84f4d636bdce73c31c420ffb29340ec79a17a863fe0778a86d727549
Jul 28 01:26:13.336715 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:13.336698    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c607d9_7b10_4b44_88ee_e81ada7ef179.slice/crio-f6566b53c24961a0a14a13a44635dcbdc495ff3323b1446fc5e4fd3657016614 WatchSource:0}: Error finding container f6566b53c24961a0a14a13a44635dcbdc495ff3323b1446fc5e4fd3657016614: Status 404 returned error can't find the container with id f6566b53c24961a0a14a13a44635dcbdc495ff3323b1446fc5e4fd3657016614
Jul 28 01:26:13.338377 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:13.338041    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89eecf33_ed8e_41fe_8fce_48a72361f9ad.slice/crio-725f71f0fec22055f1741dc021c1225ce3b8813f0cfa146d39595d5eabc48aa4 WatchSource:0}: Error finding container 725f71f0fec22055f1741dc021c1225ce3b8813f0cfa146d39595d5eabc48aa4: Status 404 returned error can't find the container with id 725f71f0fec22055f1741dc021c1225ce3b8813f0cfa146d39595d5eabc48aa4
Jul 28 01:26:13.438645 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:13.438430    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal" event={"ID":"088df6a04689e2bf1edce0757bca5189","Type":"ContainerStarted","Data":"fef9fb74c36db9372a71867948161f38ea9f48ccf1c40b9a8c657293a9c1a71e"}
Jul 28 01:26:13.440981 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:13.440950    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jbm5x" event={"ID":"bc143eaf-873e-484b-9ef0-adc12e645c43","Type":"ContainerStarted","Data":"13c9cf2c9a974b43b92a39bd45a0992c5e0fe6414ceee428a5bd092a78027faa"}
Jul 28 01:26:13.444118 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:13.444094    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm" event={"ID":"61c607d9-7b10-4b44-88ee-e81ada7ef179","Type":"ContainerStarted","Data":"f6566b53c24961a0a14a13a44635dcbdc495ff3323b1446fc5e4fd3657016614"}
Jul 28 01:26:13.447538 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:13.447515    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p4bgp" event={"ID":"d356f315-756f-439b-882e-35320cae4a10","Type":"ContainerStarted","Data":"c5d88f7d84f4d636bdce73c31c420ffb29340ec79a17a863fe0778a86d727549"}
Jul 28 01:26:13.452140 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:13.452107    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cdqxp" event={"ID":"3aa211a5-d538-4fd8-a415-29ea6de08067","Type":"ContainerStarted","Data":"364afef3e8b0f325e4bc04e12d10d85f45c1a23aa7141fc022b5c39df1df5531"}
Jul 28 01:26:13.455096 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:13.455077    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwgbx" event={"ID":"4db16d2d-9789-4546-96e8-3438dd7fdba8","Type":"ContainerStarted","Data":"d4cec1ec48c52b3dd08c1991c4c2f31161feb62bb7ffe9fce47ddd413273e9ff"}
Jul 28 01:26:13.456208 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:13.456184    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jh4fs" event={"ID":"08fb009b-b1f6-465f-b258-7e408d59d0d4","Type":"ContainerStarted","Data":"2dd9b0d567d2accdd00dc8030193e5df23af76f2e4f6ce3208590e1a3d9ee027"}
Jul 28 01:26:13.457500 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:13.457474    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2h5rf" event={"ID":"89eecf33-ed8e-41fe-8fce-48a72361f9ad","Type":"ContainerStarted","Data":"725f71f0fec22055f1741dc021c1225ce3b8813f0cfa146d39595d5eabc48aa4"}
Jul 28 01:26:13.458809 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:13.458783    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q2vqt" event={"ID":"45a5d7f9-5b74-4f84-a448-d89cb7445b23","Type":"ContainerStarted","Data":"e16741ce414f4a99d7b83253a138e3869af3ad5ed1eba2df40d4045e50cde096"}
Jul 28 01:26:13.461255 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:13.461236    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6" event={"ID":"49687516-e291-414c-95b7-9f319406a20d","Type":"ContainerStarted","Data":"80f44a3bbbc75bf8ceafd806f7201fc2394f876474fa1955693cafcc80c93835"}
Jul 28 01:26:13.855507 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:13.855372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:13.855666 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:13.855560    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:13.855666 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:13.855632    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret podName:be103d49-24c3-4a2e-9a55-2fcc960c1a71 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:15.855611637 +0000 UTC m=+6.026650667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret") pod "global-pull-secret-syncer-nf7cb" (UID: "be103d49-24c3-4a2e-9a55-2fcc960c1a71") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:13.855796 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:13.855669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs\") pod \"network-metrics-daemon-22fhh\" (UID: \"8849435f-9f9d-4fd2-858c-b037077d839c\") " pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:13.855796 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:13.855794    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:13.855895 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:13.855835    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs podName:8849435f-9f9d-4fd2-858c-b037077d839c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:15.855820493 +0000 UTC m=+6.026859511 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs") pod "network-metrics-daemon-22fhh" (UID: "8849435f-9f9d-4fd2-858c-b037077d839c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:13.956451 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:13.956413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqkct\" (UniqueName: \"kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct\") pod \"network-check-target-km9pg\" (UID: \"14745d0d-a327-4289-84d7-213c1e2f3eef\") " pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:13.956732 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:13.956660    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:13.956732 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:13.956686    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:13.956732 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:13.956706    2576 projected.go:196] Error preparing data for projected volume kube-api-access-hqkct for pod openshift-network-diagnostics/network-check-target-km9pg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:13.957001 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:13.956785    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct podName:14745d0d-a327-4289-84d7-213c1e2f3eef nodeName:}" failed. No retries permitted until 2026-07-28 01:26:15.956765861 +0000 UTC m=+6.127804891 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqkct" (UniqueName: "kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct") pod "network-check-target-km9pg" (UID: "14745d0d-a327-4289-84d7-213c1e2f3eef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:14.331035 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:14.330986    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:21:12 +0000 UTC" deadline="2028-04-03 18:41:42.353057876 +0000 UTC"
Jul 28 01:26:14.331035 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:14.331034    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14777h15m28.022028433s"
Jul 28 01:26:14.429728 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:14.428942    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:14.429728 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:14.429070    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:14.429728 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:14.429151    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:14.429728 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:14.429220    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:14.429728 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:14.429278    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:14.429728 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:14.429358    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:14.475863 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:14.475676    2576 generic.go:356] "Generic (PLEG): container finished" podID="a2e5a9fa29df8176300d2ac7d8254fea" containerID="f51788419a84a7188a78f9fd6a04b2fd8a4de9ae3777b6f876ced9821bd7812b" exitCode=0
Jul 28 01:26:14.476283 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:14.476241    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal" event={"ID":"a2e5a9fa29df8176300d2ac7d8254fea","Type":"ContainerDied","Data":"f51788419a84a7188a78f9fd6a04b2fd8a4de9ae3777b6f876ced9821bd7812b"}
Jul 28 01:26:14.494073 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:14.493754    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-168.ec2.internal" podStartSLOduration=3.492962049 podStartE2EDuration="3.492962049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:13.452537008 +0000 UTC m=+3.623576046" watchObservedRunningTime="2026-07-28 01:26:14.492962049 +0000 UTC m=+4.664001088"
Jul 28 01:26:15.487668 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:15.487627    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal" event={"ID":"a2e5a9fa29df8176300d2ac7d8254fea","Type":"ContainerStarted","Data":"63a8ea770ea7dd94b144a478ccf7eac20ae14c68ef14373984556e5172c1f221"}
Jul 28 01:26:15.504095 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:15.504035    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-168.ec2.internal" podStartSLOduration=4.504013277 podStartE2EDuration="4.504013277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:15.50307707 +0000 UTC m=+5.674116108" watchObservedRunningTime="2026-07-28 01:26:15.504013277 +0000 UTC m=+5.675052318"
Jul 28 01:26:15.871002 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:15.870917    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:15.871002 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:15.870991    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs\") pod \"network-metrics-daemon-22fhh\" (UID: \"8849435f-9f9d-4fd2-858c-b037077d839c\") " pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:15.871223 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:15.871113    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:15.871223 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:15.871171    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs podName:8849435f-9f9d-4fd2-858c-b037077d839c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:19.871153954 +0000 UTC m=+10.042192987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs") pod "network-metrics-daemon-22fhh" (UID: "8849435f-9f9d-4fd2-858c-b037077d839c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:15.871636 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:15.871596    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:15.871736 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:15.871653    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret podName:be103d49-24c3-4a2e-9a55-2fcc960c1a71 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:19.871637608 +0000 UTC m=+10.042676636 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret") pod "global-pull-secret-syncer-nf7cb" (UID: "be103d49-24c3-4a2e-9a55-2fcc960c1a71") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:15.972192 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:15.972153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqkct\" (UniqueName: \"kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct\") pod \"network-check-target-km9pg\" (UID: \"14745d0d-a327-4289-84d7-213c1e2f3eef\") " pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:15.972422 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:15.972402    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:15.972507 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:15.972430    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:15.972507 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:15.972446    2576 projected.go:196] Error preparing data for projected volume kube-api-access-hqkct for pod openshift-network-diagnostics/network-check-target-km9pg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:15.972600 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:15.972507    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct podName:14745d0d-a327-4289-84d7-213c1e2f3eef nodeName:}" failed. No retries permitted until 2026-07-28 01:26:19.972489525 +0000 UTC m=+10.143528563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqkct" (UniqueName: "kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct") pod "network-check-target-km9pg" (UID: "14745d0d-a327-4289-84d7-213c1e2f3eef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:16.427408 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:16.426044    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:16.427408 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:16.426174    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:16.427408 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:16.426575    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:16.427408 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:16.426669    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:16.427408 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:16.426736    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:16.427408 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:16.426857    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:18.430056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:18.430025    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:18.430523 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:18.430069    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:18.430523 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:18.430176    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:18.430523 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:18.430228    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:18.430523 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:18.430333    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:18.430746 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:18.430526    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:19.901768 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:19.901727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs\") pod \"network-metrics-daemon-22fhh\" (UID: \"8849435f-9f9d-4fd2-858c-b037077d839c\") " pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:19.902229 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:19.901810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:19.902229 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:19.901937    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:19.902229 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:19.902010    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret podName:be103d49-24c3-4a2e-9a55-2fcc960c1a71 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:27.901990587 +0000 UTC m=+18.073029604 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret") pod "global-pull-secret-syncer-nf7cb" (UID: "be103d49-24c3-4a2e-9a55-2fcc960c1a71") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:19.902494 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:19.902470    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:19.902563 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:19.902554    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs podName:8849435f-9f9d-4fd2-858c-b037077d839c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:27.90253734 +0000 UTC m=+18.073576358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs") pod "network-metrics-daemon-22fhh" (UID: "8849435f-9f9d-4fd2-858c-b037077d839c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:20.002989 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:20.002955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqkct\" (UniqueName: \"kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct\") pod \"network-check-target-km9pg\" (UID: \"14745d0d-a327-4289-84d7-213c1e2f3eef\") " pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:20.003176 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:20.003116    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:20.003176 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:20.003139    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:20.003176 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:20.003153    2576 projected.go:196] Error preparing data for projected volume kube-api-access-hqkct for pod openshift-network-diagnostics/network-check-target-km9pg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:20.003337 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:20.003214    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct podName:14745d0d-a327-4289-84d7-213c1e2f3eef nodeName:}" failed. No retries permitted until 2026-07-28 01:26:28.003196799 +0000 UTC m=+18.174235814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqkct" (UniqueName: "kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct") pod "network-check-target-km9pg" (UID: "14745d0d-a327-4289-84d7-213c1e2f3eef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:20.426574 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:20.426539    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:20.426739 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:20.426639    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:20.426739 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:20.426655    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:20.426862 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:20.426768    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:20.426862 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:20.426831    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:20.426933 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:20.426913    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:22.425923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:22.425891    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:22.426353 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:22.426025    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:22.426451 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:22.426419    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:22.427412 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:22.426532    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:22.427412 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:22.426663    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:22.427412 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:22.426745    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:24.429273 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:24.429243    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:24.429645 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:24.429243    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:24.429645 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:24.429350    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:24.429645 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:24.429243    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:24.429645 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:24.429429    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:24.429645 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:24.429519    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:26.425999 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:26.425952    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:26.425999 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:26.426001    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:26.426499 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:26.425960    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:26.426499 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:26.426097    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:26.426499 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:26.426211    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:26.426499 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:26.426313    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:27.965069 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:27.965023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs\") pod \"network-metrics-daemon-22fhh\" (UID: \"8849435f-9f9d-4fd2-858c-b037077d839c\") " pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:27.965620 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:27.965099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:27.965620 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:27.965201    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:27.965620 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:27.965208    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:27.965620 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:27.965269    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret podName:be103d49-24c3-4a2e-9a55-2fcc960c1a71 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:43.96525165 +0000 UTC m=+34.136290670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret") pod "global-pull-secret-syncer-nf7cb" (UID: "be103d49-24c3-4a2e-9a55-2fcc960c1a71") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:27.965620 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:27.965290    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs podName:8849435f-9f9d-4fd2-858c-b037077d839c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:43.96528026 +0000 UTC m=+34.136319277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs") pod "network-metrics-daemon-22fhh" (UID: "8849435f-9f9d-4fd2-858c-b037077d839c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:28.066046 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:28.066004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqkct\" (UniqueName: \"kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct\") pod \"network-check-target-km9pg\" (UID: \"14745d0d-a327-4289-84d7-213c1e2f3eef\") " pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:28.066248 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:28.066187    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:28.066248 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:28.066216    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:28.066248 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:28.066227    2576 projected.go:196] Error preparing data for projected volume kube-api-access-hqkct for pod openshift-network-diagnostics/network-check-target-km9pg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:28.066411 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:28.066286    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct podName:14745d0d-a327-4289-84d7-213c1e2f3eef nodeName:}" failed. No retries permitted until 2026-07-28 01:26:44.066266318 +0000 UTC m=+34.237305355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqkct" (UniqueName: "kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct") pod "network-check-target-km9pg" (UID: "14745d0d-a327-4289-84d7-213c1e2f3eef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:28.426030 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:28.425992    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:28.426222 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:28.426141    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:28.426222 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:28.426150    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:28.426304 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:28.426233    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:28.426304 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:28.426269    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:28.426367 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:28.426323    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:30.426956 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:30.426920    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:30.427431 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:30.427004    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:30.427431 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:30.427038    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:30.427431 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:30.427120    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:30.427431 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:30.427152    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:30.427431 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:30.427243    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:32.426200 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.425897    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:32.426912 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.425897    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:32.426912 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:32.426296    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:32.426912 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.425978    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:32.426912 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:32.426353    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:32.426912 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:32.426438    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:32.525834 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.525766    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jbm5x" event={"ID":"bc143eaf-873e-484b-9ef0-adc12e645c43","Type":"ContainerStarted","Data":"d12060efa942e95e0ba571a7eb06372136864e4ddead67743c353f71c9fd7071"}
Jul 28 01:26:32.529323 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.529266    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm" event={"ID":"61c607d9-7b10-4b44-88ee-e81ada7ef179","Type":"ContainerStarted","Data":"04b87ae4e4dfaf3b0ae8cbc4e4b6a048be7bb46a59664e72d228739f8992ef1e"}
Jul 28 01:26:32.529455 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.529351    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm" event={"ID":"61c607d9-7b10-4b44-88ee-e81ada7ef179","Type":"ContainerStarted","Data":"5f3b0372ddbcc179e53896a0b6c493c43fe10922b2b24be1a4d33f71426ef7c3"}
Jul 28 01:26:32.529516 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.529455    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm" event={"ID":"61c607d9-7b10-4b44-88ee-e81ada7ef179","Type":"ContainerStarted","Data":"4a46d2c493aabd1604d428b6751dee9282e8a93b31b9c887bd828707022ebf57"}
Jul 28 01:26:32.529516 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.529507    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm" event={"ID":"61c607d9-7b10-4b44-88ee-e81ada7ef179","Type":"ContainerStarted","Data":"1a013895ee086bc5bea9b677aa09d23682a00e8ad4098d0a399370e4cda52e04"}
Jul 28 01:26:32.531191 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.531146    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cdqxp" event={"ID":"3aa211a5-d538-4fd8-a415-29ea6de08067","Type":"ContainerStarted","Data":"01bdda338f2348ce43ddf757478a385de420c395f3ba024b701acbb879831bb5"}
Jul 28 01:26:32.532850 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.532828    2576 generic.go:356] "Generic (PLEG): container finished" podID="4db16d2d-9789-4546-96e8-3438dd7fdba8" containerID="c7c81fd024af21c1dac3161005523de2f32deffb0f789484a33625614001bc74" exitCode=0
Jul 28 01:26:32.533055 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.533034    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwgbx" event={"ID":"4db16d2d-9789-4546-96e8-3438dd7fdba8","Type":"ContainerDied","Data":"c7c81fd024af21c1dac3161005523de2f32deffb0f789484a33625614001bc74"}
Jul 28 01:26:32.534917 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.534882    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jh4fs" event={"ID":"08fb009b-b1f6-465f-b258-7e408d59d0d4","Type":"ContainerStarted","Data":"19aa5dcfdf40681fe4863c86ea04425dabcb5ffc93cb7b26cad11a629fc75b99"}
Jul 28 01:26:32.536432 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.536409    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2h5rf" event={"ID":"89eecf33-ed8e-41fe-8fce-48a72361f9ad","Type":"ContainerStarted","Data":"45a1c9fd5ff507ec39a6438c7de52411c6f416b75bce77511f7f9ec5f6748b88"}
Jul 28 01:26:32.538202 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.538180    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q2vqt" event={"ID":"45a5d7f9-5b74-4f84-a448-d89cb7445b23","Type":"ContainerStarted","Data":"4ebc336725db969efa712e56ddd6974528b8dcfdbeb35db15b9f937d5a41a1d5"}
Jul 28 01:26:32.539907 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.539889    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6" event={"ID":"49687516-e291-414c-95b7-9f319406a20d","Type":"ContainerStarted","Data":"4b957214c09fe3626733cda269b1cbd72fe0e9aaa47fcba000274b9dbe8b6861"}
Jul 28 01:26:32.541219 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.541175    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jbm5x" podStartSLOduration=4.106885869 podStartE2EDuration="22.541158814s" totalImagesPullingTime="18.434272945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:32.540590092 +0000 UTC m=+22.711629130" watchObservedRunningTime="2026-07-28 01:26:32.541158814 +0000 UTC m=+22.712197853"
Jul 28 01:26:32.579849 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.578529    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2h5rf" podStartSLOduration=4.0965946540000004 podStartE2EDuration="22.57851172s" totalImagesPullingTime="18.481917066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:32.577745493 +0000 UTC m=+22.748784531" watchObservedRunningTime="2026-07-28 01:26:32.57851172 +0000 UTC m=+22.749550758"
Jul 28 01:26:32.593459 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.593417    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jh4fs" podStartSLOduration=4.159548945 podStartE2EDuration="22.593377357s" totalImagesPullingTime="18.433828412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:32.593339698 +0000 UTC m=+22.764378733" watchObservedRunningTime="2026-07-28 01:26:32.593377357 +0000 UTC m=+22.764416395"
Jul 28 01:26:32.614458 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.614417    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cdqxp" podStartSLOduration=11.979595777 podStartE2EDuration="22.614405759s" totalImagesPullingTime="10.634809982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:32.61426996 +0000 UTC m=+22.785308996" watchObservedRunningTime="2026-07-28 01:26:32.614405759 +0000 UTC m=+22.785444787"
Jul 28 01:26:32.627603 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.627582    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cdqxp"
Jul 28 01:26:32.628178 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.628163    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cdqxp"
Jul 28 01:26:32.630291 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:32.630260    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q2vqt" podStartSLOduration=4.188262016 podStartE2EDuration="22.630249209s" totalImagesPullingTime="18.441987193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:32.629890567 +0000 UTC m=+22.800929603" watchObservedRunningTime="2026-07-28 01:26:32.630249209 +0000 UTC m=+22.801288246"
Jul 28 01:26:33.183336 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:33.183308    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 01:26:33.306588 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:33.306470    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T01:26:33.183331853Z","UUID":"7d1c7cb6-fde0-4040-9da2-1a65af624a52","Handler":null,"Name":"","Endpoint":""}
Jul 28 01:26:33.308263 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:33.308242    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 01:26:33.308377 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:33.308284    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 01:26:33.544646 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:33.544609    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6" event={"ID":"49687516-e291-414c-95b7-9f319406a20d","Type":"ContainerStarted","Data":"0bf66d8e5713a5cc40b6780f37d5e03a30c9cd5e9c003dc25318596e028f3289"}
Jul 28 01:26:33.547937 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:33.547907    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm" event={"ID":"61c607d9-7b10-4b44-88ee-e81ada7ef179","Type":"ContainerStarted","Data":"ae22c0440d5ce01f36f2c801d21b347dd098af2505cfeebe5d46dc8ddedd08c2"}
Jul 28 01:26:33.548059 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:33.547992    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm" event={"ID":"61c607d9-7b10-4b44-88ee-e81ada7ef179","Type":"ContainerStarted","Data":"e06275c139faac4dd21a0df3ebf58571e97551ff0fefa7c3ecf7d41df92002b0"}
Jul 28 01:26:33.549616 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:33.549588    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p4bgp" event={"ID":"d356f315-756f-439b-882e-35320cae4a10","Type":"ContainerStarted","Data":"220a354f438948fa85fbeb56128d0862d269428f265fba229c153e97b1d82088"}
Jul 28 01:26:33.550497 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:33.550474    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cdqxp"
Jul 28 01:26:33.550934 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:33.550917    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cdqxp"
Jul 28 01:26:33.566602 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:33.566552    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p4bgp" podStartSLOduration=5.128896832 podStartE2EDuration="23.566533525s" totalImagesPullingTime="18.437636693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:33.565920337 +0000 UTC m=+23.736959375" watchObservedRunningTime="2026-07-28 01:26:33.566533525 +0000 UTC m=+23.737572564"
Jul 28 01:26:34.426361 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:34.426327    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:34.426629 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:34.426326    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:34.426629 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:34.426475    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:34.426629 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:34.426474    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:34.426629 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:34.426576    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:34.426864 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:34.426662    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:34.555027 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:34.554924    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6" event={"ID":"49687516-e291-414c-95b7-9f319406a20d","Type":"ContainerStarted","Data":"c5e2a5ab595b3c47a00e00259752931a911ef9e74d445d8ad45bd96963c726e8"}
Jul 28 01:26:34.574500 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:34.574442    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2gb6" podStartSLOduration=3.915382214 podStartE2EDuration="24.574427603s" totalImagesPullingTime="20.659045389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:34.57400654 +0000 UTC m=+24.745045576" watchObservedRunningTime="2026-07-28 01:26:34.574427603 +0000 UTC m=+24.745466639"
Jul 28 01:26:35.560759 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:35.560717    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm" event={"ID":"61c607d9-7b10-4b44-88ee-e81ada7ef179","Type":"ContainerStarted","Data":"c433c6135f95b0f463cd76c515b1c9a3d5ece8c01569fb74546b57e2aee06592"}
Jul 28 01:26:36.426556 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:36.426525    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:36.426732 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:36.426525    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:36.426732 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:36.426675    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:36.426833 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:36.426525    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:36.426833 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:36.426748    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:36.426934 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:36.426823    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:37.569698 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:37.569433    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm" event={"ID":"61c607d9-7b10-4b44-88ee-e81ada7ef179","Type":"ContainerStarted","Data":"a2572abbb50714c892d7072f97dc7239d16b105b9f85800a59789e09cc0acfb6"}
Jul 28 01:26:37.570543 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:37.569750    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:37.570543 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:37.569798    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:37.571334 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:37.571301    2576 generic.go:356] "Generic (PLEG): container finished" podID="4db16d2d-9789-4546-96e8-3438dd7fdba8" containerID="c89f5423b94b62ae5e8854c78f57f53e6c34917baf858c0b723ba95eb8ac5be0" exitCode=0
Jul 28 01:26:37.571454 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:37.571351    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwgbx" event={"ID":"4db16d2d-9789-4546-96e8-3438dd7fdba8","Type":"ContainerDied","Data":"c89f5423b94b62ae5e8854c78f57f53e6c34917baf858c0b723ba95eb8ac5be0"}
Jul 28 01:26:37.586559 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:37.586537    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:37.597522 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:37.597483    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm" podStartSLOduration=9.075672705 podStartE2EDuration="27.597469699s" totalImagesPullingTime="18.521796994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:37.597312765 +0000 UTC m=+27.768351806" watchObservedRunningTime="2026-07-28 01:26:37.597469699 +0000 UTC m=+27.768508738"
Jul 28 01:26:38.430160 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:38.429884    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:38.430363 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:38.429900    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:38.430363 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:38.430219    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:38.430363 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:38.429913    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:38.430363 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:38.430316    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:38.430597 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:38.430416    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:38.573744 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:38.573716    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:38.588607 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:38.588577    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:26:38.994300 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:38.994267    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-km9pg"]
Jul 28 01:26:38.994499 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:38.994404    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:38.994555 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:38.994519    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:38.997992 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:38.997965    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-22fhh"]
Jul 28 01:26:38.998141 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:38.998080    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:38.998226 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:38.998205    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:38.998834 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:38.998806    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nf7cb"]
Jul 28 01:26:38.998950 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:38.998897    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:38.998999 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:38.998977    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:39.577563 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:39.577528    2576 generic.go:356] "Generic (PLEG): container finished" podID="4db16d2d-9789-4546-96e8-3438dd7fdba8" containerID="88ba29d8a539135bae546070e2f92ae310041ef23c1d3930025bcb3137aa71da" exitCode=0
Jul 28 01:26:39.578131 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:39.577613    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwgbx" event={"ID":"4db16d2d-9789-4546-96e8-3438dd7fdba8","Type":"ContainerDied","Data":"88ba29d8a539135bae546070e2f92ae310041ef23c1d3930025bcb3137aa71da"}
Jul 28 01:26:40.391600 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:40.391562    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:40.428730 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:40.428698    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:40.428879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:40.428740    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:40.428879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:40.428741    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:40.428879 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:40.428809    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:40.428987 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:40.428897    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:40.428987 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:40.428970    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:40.518012 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:40.517984    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:40.835101 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:40.834893    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:41.399950 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:41.399920    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:41.500529 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:41.500502    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:41.587813 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:41.587757    2576 generic.go:356] "Generic (PLEG): container finished" podID="4db16d2d-9789-4546-96e8-3438dd7fdba8" containerID="3093ca779f99182a22eaab38975ea64653164087f88bd67aba710635f285959b" exitCode=0
Jul 28 01:26:41.587813 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:41.587810    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwgbx" event={"ID":"4db16d2d-9789-4546-96e8-3438dd7fdba8","Type":"ContainerDied","Data":"3093ca779f99182a22eaab38975ea64653164087f88bd67aba710635f285959b"}
Jul 28 01:26:42.429752 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:42.429721    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:42.429752 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:42.429748    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:42.429752 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:42.429754    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:42.430592 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:42.429835    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:42.430592 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:42.429957    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:42.430592 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:42.430046    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:43.983993 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:43.983958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:43.984481 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:43.984013    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs\") pod \"network-metrics-daemon-22fhh\" (UID: \"8849435f-9f9d-4fd2-858c-b037077d839c\") " pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:43.984481 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:43.984117    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:43.984481 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:43.984148    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:43.984481 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:43.984183    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret podName:be103d49-24c3-4a2e-9a55-2fcc960c1a71 nodeName:}" failed. No retries permitted until 2026-07-28 01:27:15.984162151 +0000 UTC m=+66.155201166 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret") pod "global-pull-secret-syncer-nf7cb" (UID: "be103d49-24c3-4a2e-9a55-2fcc960c1a71") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:43.984481 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:43.984203    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs podName:8849435f-9f9d-4fd2-858c-b037077d839c nodeName:}" failed. No retries permitted until 2026-07-28 01:27:15.984194568 +0000 UTC m=+66.155233585 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs") pod "network-metrics-daemon-22fhh" (UID: "8849435f-9f9d-4fd2-858c-b037077d839c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:44.084661 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.084623    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqkct\" (UniqueName: \"kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct\") pod \"network-check-target-km9pg\" (UID: \"14745d0d-a327-4289-84d7-213c1e2f3eef\") " pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:44.084833 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:44.084782    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:44.084833 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:44.084811    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:44.084833 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:44.084821    2576 projected.go:196] Error preparing data for projected volume kube-api-access-hqkct for pod openshift-network-diagnostics/network-check-target-km9pg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:44.084964 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:44.084884    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct podName:14745d0d-a327-4289-84d7-213c1e2f3eef nodeName:}" failed. No retries permitted until 2026-07-28 01:27:16.084866512 +0000 UTC m=+66.255905527 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqkct" (UniqueName: "kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct") pod "network-check-target-km9pg" (UID: "14745d0d-a327-4289-84d7-213c1e2f3eef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:44.425822 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.425782    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:44.425995 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.425783    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:44.425995 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:44.425916    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nf7cb" podUID="be103d49-24c3-4a2e-9a55-2fcc960c1a71"
Jul 28 01:26:44.426111 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:44.426010    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22fhh" podUID="8849435f-9f9d-4fd2-858c-b037077d839c"
Jul 28 01:26:44.426111 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.425787    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:44.426111 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:44.426093    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km9pg" podUID="14745d0d-a327-4289-84d7-213c1e2f3eef"
Jul 28 01:26:44.606460 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.606368    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-168.ec2.internal" event="NodeReady"
Jul 28 01:26:44.606604 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.606551    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 01:26:44.656099 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.656067    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cj2zl"]
Jul 28 01:26:44.658577 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.658553    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:44.660979 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.660955    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 01:26:44.661260 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.661233    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 01:26:44.661456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.661425    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7zgnb\""
Jul 28 01:26:44.664890 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.664867    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6zpwr"]
Jul 28 01:26:44.667895 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.667875    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6zpwr"
Jul 28 01:26:44.670470 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.670453    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 01:26:44.670992 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.670709    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-x7c4m\""
Jul 28 01:26:44.670992 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.670738    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 01:26:44.670992 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.670843    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 01:26:44.673410 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.673363    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cj2zl"]
Jul 28 01:26:44.681548 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.681521    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6zpwr"]
Jul 28 01:26:44.789125 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.789082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f01ecde2-601f-4327-b1d6-43b8b46fa2d5-cert\") pod \"ingress-canary-6zpwr\" (UID: \"f01ecde2-601f-4327-b1d6-43b8b46fa2d5\") " pod="openshift-ingress-canary/ingress-canary-6zpwr"
Jul 28 01:26:44.789307 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.789145    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2960b4e1-5587-486f-95b3-aa18155fd6fe-config-volume\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:44.789307 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.789224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2960b4e1-5587-486f-95b3-aa18155fd6fe-tmp-dir\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:44.789417 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.789303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wz4g\" (UniqueName: \"kubernetes.io/projected/2960b4e1-5587-486f-95b3-aa18155fd6fe-kube-api-access-7wz4g\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:44.789417 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.789333    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:44.789417 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.789408    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvnmk\" (UniqueName: \"kubernetes.io/projected/f01ecde2-601f-4327-b1d6-43b8b46fa2d5-kube-api-access-wvnmk\") pod \"ingress-canary-6zpwr\" (UID: \"f01ecde2-601f-4327-b1d6-43b8b46fa2d5\") " pod="openshift-ingress-canary/ingress-canary-6zpwr"
Jul 28 01:26:44.890004 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.889923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvnmk\" (UniqueName: \"kubernetes.io/projected/f01ecde2-601f-4327-b1d6-43b8b46fa2d5-kube-api-access-wvnmk\") pod \"ingress-canary-6zpwr\" (UID: \"f01ecde2-601f-4327-b1d6-43b8b46fa2d5\") " pod="openshift-ingress-canary/ingress-canary-6zpwr"
Jul 28 01:26:44.890004 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.889967    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f01ecde2-601f-4327-b1d6-43b8b46fa2d5-cert\") pod \"ingress-canary-6zpwr\" (UID: \"f01ecde2-601f-4327-b1d6-43b8b46fa2d5\") " pod="openshift-ingress-canary/ingress-canary-6zpwr"
Jul 28 01:26:44.890226 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.890117    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2960b4e1-5587-486f-95b3-aa18155fd6fe-config-volume\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:44.890226 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.890166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2960b4e1-5587-486f-95b3-aa18155fd6fe-tmp-dir\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:44.890226 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.890213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wz4g\" (UniqueName: \"kubernetes.io/projected/2960b4e1-5587-486f-95b3-aa18155fd6fe-kube-api-access-7wz4g\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:44.890465 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.890244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:44.890465 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:44.890351    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:44.890465 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:44.890439    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls podName:2960b4e1-5587-486f-95b3-aa18155fd6fe nodeName:}" failed. No retries permitted until 2026-07-28 01:26:45.390419397 +0000 UTC m=+35.561458435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls") pod "dns-default-cj2zl" (UID: "2960b4e1-5587-486f-95b3-aa18155fd6fe") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:44.890619 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.890558    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2960b4e1-5587-486f-95b3-aa18155fd6fe-tmp-dir\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:44.890915 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.890881    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2960b4e1-5587-486f-95b3-aa18155fd6fe-config-volume\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:44.894847 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.894824    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f01ecde2-601f-4327-b1d6-43b8b46fa2d5-cert\") pod \"ingress-canary-6zpwr\" (UID: \"f01ecde2-601f-4327-b1d6-43b8b46fa2d5\") " pod="openshift-ingress-canary/ingress-canary-6zpwr"
Jul 28 01:26:44.912172 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.912140    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvnmk\" (UniqueName: \"kubernetes.io/projected/f01ecde2-601f-4327-b1d6-43b8b46fa2d5-kube-api-access-wvnmk\") pod \"ingress-canary-6zpwr\" (UID: \"f01ecde2-601f-4327-b1d6-43b8b46fa2d5\") " pod="openshift-ingress-canary/ingress-canary-6zpwr"
Jul 28 01:26:44.912172 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.912163    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wz4g\" (UniqueName: \"kubernetes.io/projected/2960b4e1-5587-486f-95b3-aa18155fd6fe-kube-api-access-7wz4g\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:44.982850 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:44.982815    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6zpwr"
Jul 28 01:26:45.394995 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:45.394958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:45.395756 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:45.395133    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:45.395756 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:45.395226    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls podName:2960b4e1-5587-486f-95b3-aa18155fd6fe nodeName:}" failed. No retries permitted until 2026-07-28 01:26:46.395205873 +0000 UTC m=+36.566244889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls") pod "dns-default-cj2zl" (UID: "2960b4e1-5587-486f-95b3-aa18155fd6fe") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:45.658893 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:45.658739    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6zpwr"]
Jul 28 01:26:45.662220 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:45.662195    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01ecde2_601f_4327_b1d6_43b8b46fa2d5.slice/crio-b54135d1f89e9d4eba52a2a05a1ca3a3c8ad7cd6a49ea0516b6325454cbdd564 WatchSource:0}: Error finding container b54135d1f89e9d4eba52a2a05a1ca3a3c8ad7cd6a49ea0516b6325454cbdd564: Status 404 returned error can't find the container with id b54135d1f89e9d4eba52a2a05a1ca3a3c8ad7cd6a49ea0516b6325454cbdd564
Jul 28 01:26:45.901986 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:45.901905    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:45.964710 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:45.964683    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:46.404715 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:46.404679    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:46.405101 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:46.404822    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:46.405101 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:46.404910    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls podName:2960b4e1-5587-486f-95b3-aa18155fd6fe nodeName:}" failed. No retries permitted until 2026-07-28 01:26:48.404896023 +0000 UTC m=+38.575935038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls") pod "dns-default-cj2zl" (UID: "2960b4e1-5587-486f-95b3-aa18155fd6fe") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:46.425847 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:46.425807    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:26:46.425847 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:46.425836    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:26:46.426089 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:46.425856    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:26:46.429839 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:46.429813    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 01:26:46.429839 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:46.429838    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 01:26:46.430041 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:46.430029    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 01:26:46.430111 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:46.430089    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 01:26:46.430307 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:46.430293    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cq5js\""
Jul 28 01:26:46.430365 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:46.430312    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lkkxh\""
Jul 28 01:26:46.603233 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:46.603200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6zpwr" event={"ID":"f01ecde2-601f-4327-b1d6-43b8b46fa2d5","Type":"ContainerStarted","Data":"b54135d1f89e9d4eba52a2a05a1ca3a3c8ad7cd6a49ea0516b6325454cbdd564"}
Jul 28 01:26:46.606256 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:46.606223    2576 generic.go:356] "Generic (PLEG): container finished" podID="4db16d2d-9789-4546-96e8-3438dd7fdba8" containerID="4f0fa8f02fae146bc61cef3765c3702087ffd13b09c5ea36d8570c0f0bfa8a97" exitCode=0
Jul 28 01:26:46.606361 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:46.606281    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwgbx" event={"ID":"4db16d2d-9789-4546-96e8-3438dd7fdba8","Type":"ContainerDied","Data":"4f0fa8f02fae146bc61cef3765c3702087ffd13b09c5ea36d8570c0f0bfa8a97"}
Jul 28 01:26:47.264120 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.264080    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5vv8g"]
Jul 28 01:26:47.266963 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.266939    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:47.269822 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.269783    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 01:26:47.270035 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.270019    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 01:26:47.270836 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.270816    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-spb28\""
Jul 28 01:26:47.271075 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.271060    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 01:26:47.271309 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.271294    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 01:26:47.284081 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.284049    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5vv8g"]
Jul 28 01:26:47.327232 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.327197    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:47.409648 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.409612    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:47.411578 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.411549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a78712d7-20fe-49b8-94c9-96ac63bd9015-data-volume\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:47.411701 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.411601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:47.411701 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.411639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:47.411701 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.411690    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmvlx\" (UniqueName: \"kubernetes.io/projected/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-api-access-kmvlx\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:47.411826 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.411709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a78712d7-20fe-49b8-94c9-96ac63bd9015-crio-socket\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:47.486255 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.486215    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:47.512679 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.512646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmvlx\" (UniqueName: \"kubernetes.io/projected/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-api-access-kmvlx\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:47.512867 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.512687    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a78712d7-20fe-49b8-94c9-96ac63bd9015-crio-socket\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:47.512867 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.512741    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a78712d7-20fe-49b8-94c9-96ac63bd9015-data-volume\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:47.512867 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.512779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:47.512867 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.512806    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:47.513137 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:47.512969    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:47.513137 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.512975    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a78712d7-20fe-49b8-94c9-96ac63bd9015-crio-socket\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:47.513137 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:47.513025    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls podName:a78712d7-20fe-49b8-94c9-96ac63bd9015 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:48.013007509 +0000 UTC m=+38.184046524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5vv8g" (UID: "a78712d7-20fe-49b8-94c9-96ac63bd9015") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:47.513137 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:47.513036    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:47.513137 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:47.513077    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-rbac-proxy-cm podName:a78712d7-20fe-49b8-94c9-96ac63bd9015 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:48.013065494 +0000 UTC m=+38.184104512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5vv8g" (UID: "a78712d7-20fe-49b8-94c9-96ac63bd9015") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:47.513427 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.513141    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a78712d7-20fe-49b8-94c9-96ac63bd9015-data-volume\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:47.523778 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.523716    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmvlx\" (UniqueName: \"kubernetes.io/projected/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-api-access-kmvlx\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:47.613431 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.613393    2576 generic.go:356] "Generic (PLEG): container finished" podID="4db16d2d-9789-4546-96e8-3438dd7fdba8" containerID="293a32a4f9c7dfe7d649fbe4dc7d37eec3389dcf9457e7e0cfb9a2f3b2a5f990" exitCode=0
Jul 28 01:26:47.613592 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.613454    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwgbx" event={"ID":"4db16d2d-9789-4546-96e8-3438dd7fdba8","Type":"ContainerDied","Data":"293a32a4f9c7dfe7d649fbe4dc7d37eec3389dcf9457e7e0cfb9a2f3b2a5f990"}
Jul 28 01:26:47.649427 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.649375    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:47.732651 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.732624    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:47.950199 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:47.950005    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:48.017659 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:48.017632    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:48.017755 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:48.017720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:48.017755 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:48.017746    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:48.017823 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:48.017808    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-rbac-proxy-cm podName:a78712d7-20fe-49b8-94c9-96ac63bd9015 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:49.017791804 +0000 UTC m=+39.188830821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5vv8g" (UID: "a78712d7-20fe-49b8-94c9-96ac63bd9015") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:48.017879 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:48.017861    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:48.017929 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:48.017919    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls podName:a78712d7-20fe-49b8-94c9-96ac63bd9015 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:49.017906389 +0000 UTC m=+39.188945404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5vv8g" (UID: "a78712d7-20fe-49b8-94c9-96ac63bd9015") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:48.076163 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:48.076138    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:48.190882 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:48.190852    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:48.323834 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:48.323763    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:48.421451 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:48.421410    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:48.421996 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:48.421561    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:48.421996 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:48.421630    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls podName:2960b4e1-5587-486f-95b3-aa18155fd6fe nodeName:}" failed. No retries permitted until 2026-07-28 01:26:52.421613698 +0000 UTC m=+42.592652713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls") pod "dns-default-cj2zl" (UID: "2960b4e1-5587-486f-95b3-aa18155fd6fe") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:48.619077 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:48.618990    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwgbx" event={"ID":"4db16d2d-9789-4546-96e8-3438dd7fdba8","Type":"ContainerStarted","Data":"a3889a3e51a9b971ed4ae6c031d786c5a7d772f0ba512b33b08a559c0c27c4f9"}
Jul 28 01:26:48.620294 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:48.620270    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6zpwr" event={"ID":"f01ecde2-601f-4327-b1d6-43b8b46fa2d5","Type":"ContainerStarted","Data":"ab5117e10278432f7fe88bad5b93dc5a11f203bc6aff767acf8950adcb88e0ba"}
Jul 28 01:26:48.651447 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:48.651369    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kwgbx" podStartSLOduration=9.913499238 podStartE2EDuration="38.651354394s" totalImagesPullingTime="28.737855156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:48.650864649 +0000 UTC m=+38.821903686" watchObservedRunningTime="2026-07-28 01:26:48.651354394 +0000 UTC m=+38.822393430"
Jul 28 01:26:48.671277 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:48.671220    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6zpwr" podStartSLOduration=2.354808815 podStartE2EDuration="4.671206013s" totalImagesPullingTime="2.316397198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:48.670254972 +0000 UTC m=+38.841294008" watchObservedRunningTime="2026-07-28 01:26:48.671206013 +0000 UTC m=+38.842245060"
Jul 28 01:26:48.768248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:48.768214    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:48.883465 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:48.883329    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:48.977033 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:48.976999    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-hq8wj"]
Jul 28 01:26:48.999094 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:48.999070    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-hq8wj"]
Jul 28 01:26:48.999231 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:48.999162    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hq8wj"
Jul 28 01:26:49.002025 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.002002    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 01:26:49.002190 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.002134    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2nq9f\""
Jul 28 01:26:49.002394 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.002363    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 01:26:49.027144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.027110    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:49.027144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.027146    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:49.027320 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:49.027234    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:49.027320 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:49.027298    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-rbac-proxy-cm podName:a78712d7-20fe-49b8-94c9-96ac63bd9015 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:51.027279414 +0000 UTC m=+41.198318446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5vv8g" (UID: "a78712d7-20fe-49b8-94c9-96ac63bd9015") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:49.027320 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:49.027302    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:49.027456 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:49.027335    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls podName:a78712d7-20fe-49b8-94c9-96ac63bd9015 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:51.027325509 +0000 UTC m=+41.198364524 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5vv8g" (UID: "a78712d7-20fe-49b8-94c9-96ac63bd9015") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:49.044309 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.044281    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:49.128035 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.127999    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc2lx\" (UniqueName: \"kubernetes.io/projected/b199680c-03b6-4cff-b96b-2aeddfd47867-kube-api-access-nc2lx\") pod \"migrator-6865c9967-hq8wj\" (UID: \"b199680c-03b6-4cff-b96b-2aeddfd47867\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hq8wj"
Jul 28 01:26:49.136768 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.136692    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:49.228333 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.228301    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc2lx\" (UniqueName: \"kubernetes.io/projected/b199680c-03b6-4cff-b96b-2aeddfd47867-kube-api-access-nc2lx\") pod \"migrator-6865c9967-hq8wj\" (UID: \"b199680c-03b6-4cff-b96b-2aeddfd47867\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hq8wj"
Jul 28 01:26:49.239278 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.239235    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc2lx\" (UniqueName: \"kubernetes.io/projected/b199680c-03b6-4cff-b96b-2aeddfd47867-kube-api-access-nc2lx\") pod \"migrator-6865c9967-hq8wj\" (UID: \"b199680c-03b6-4cff-b96b-2aeddfd47867\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hq8wj"
Jul 28 01:26:49.308064 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.308026    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hq8wj"
Jul 28 01:26:49.448568 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.448421    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jbm5x_bc143eaf-873e-484b-9ef0-adc12e645c43/dns-node-resolver/0.log"
Jul 28 01:26:49.452044 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.452011    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-hq8wj"]
Jul 28 01:26:49.459152 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:26:49.459124    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb199680c_03b6_4cff_b96b_2aeddfd47867.slice/crio-1cbf3e3f9a2651f4a4297bc242aff3f248b3a320e2fd6bb59ea519561bc3a1e7 WatchSource:0}: Error finding container 1cbf3e3f9a2651f4a4297bc242aff3f248b3a320e2fd6bb59ea519561bc3a1e7: Status 404 returned error can't find the container with id 1cbf3e3f9a2651f4a4297bc242aff3f248b3a320e2fd6bb59ea519561bc3a1e7
Jul 28 01:26:49.624637 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.624598    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hq8wj" event={"ID":"b199680c-03b6-4cff-b96b-2aeddfd47867","Type":"ContainerStarted","Data":"1cbf3e3f9a2651f4a4297bc242aff3f248b3a320e2fd6bb59ea519561bc3a1e7"}
Jul 28 01:26:49.676279 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.676199    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:49.808006 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.807973    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:49.972021 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:49.971936    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:50.447636 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:50.447610    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q2vqt_45a5d7f9-5b74-4f84-a448-d89cb7445b23/node-ca/0.log"
Jul 28 01:26:51.041734 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:51.041694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:51.042093 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:51.041746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:51.042093 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:51.041940    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:51.042093 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:51.042000    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls podName:a78712d7-20fe-49b8-94c9-96ac63bd9015 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:55.041980803 +0000 UTC m=+45.213019832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5vv8g" (UID: "a78712d7-20fe-49b8-94c9-96ac63bd9015") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:51.042349 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:51.042289    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:51.042349 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:51.042349    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-rbac-proxy-cm podName:a78712d7-20fe-49b8-94c9-96ac63bd9015 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:55.042335662 +0000 UTC m=+45.213374681 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5vv8g" (UID: "a78712d7-20fe-49b8-94c9-96ac63bd9015") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:51.045870 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:51.045855    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6zpwr_f01ecde2-601f-4327-b1d6-43b8b46fa2d5/serve-healthcheck-canary/0.log"
Jul 28 01:26:51.359177 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:51.359094    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:51.630666 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:51.630579    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hq8wj" event={"ID":"b199680c-03b6-4cff-b96b-2aeddfd47867","Type":"ContainerStarted","Data":"c3bb845f7168e2c1524769bfef348718602efcb0ff960e1945e5804235dc68bd"}
Jul 28 01:26:51.630666 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:51.630629    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hq8wj" event={"ID":"b199680c-03b6-4cff-b96b-2aeddfd47867","Type":"ContainerStarted","Data":"b7e015d4b5f46384454fba8b44c2d49ecd98717c23fd83823aea45f1c73196fb"}
Jul 28 01:26:52.451279 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:52.451242    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:26:52.451707 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:52.451349    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:52.451707 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:52.451423    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls podName:2960b4e1-5587-486f-95b3-aa18155fd6fe nodeName:}" failed. No retries permitted until 2026-07-28 01:27:00.451405979 +0000 UTC m=+50.622444997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls") pod "dns-default-cj2zl" (UID: "2960b4e1-5587-486f-95b3-aa18155fd6fe") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:52.916832 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:52.916802    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:54.103975 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:54.103939    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:54.344670 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:54.344634    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:54.479492 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:54.479402    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:54.591277 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:54.591245    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:55.072571 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:55.072535    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:55.072571 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:55.072573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:55.072799 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:55.072712    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:55.072799 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:26:55.072771    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls podName:a78712d7-20fe-49b8-94c9-96ac63bd9015 nodeName:}" failed. No retries permitted until 2026-07-28 01:27:03.072755097 +0000 UTC m=+53.243794111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5vv8g" (UID: "a78712d7-20fe-49b8-94c9-96ac63bd9015") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:55.073097 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:55.073075    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a78712d7-20fe-49b8-94c9-96ac63bd9015-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:26:55.097904 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:55.097862    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:55.580730 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:55.580697    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:56.189535 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:56.189507    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:56.372792 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:56.372763    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:56.784055 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:56.784024    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:56.996301 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:56.996270    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:57.140077 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:57.139999    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:57.594493 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:57.594467    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:57.771647 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:57.771618    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:57.895108 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:57.895030    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:58.161790 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:58.161712    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:58.371242 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:58.371213    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:58.565191 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:58.565159    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:58.694095 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:58.694054    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:58.962927 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:58.962850    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:59.372057 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:59.372026    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:59.469955 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:59.469925    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:59.572443 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:59.572410    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:59.966684 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:26:59.966653    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:00.075841 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:00.075810    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:00.171367 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:00.171333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:00.511334 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:00.511290    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:27:00.514397 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:00.514366    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2960b4e1-5587-486f-95b3-aa18155fd6fe-metrics-tls\") pod \"dns-default-cj2zl\" (UID: \"2960b4e1-5587-486f-95b3-aa18155fd6fe\") " pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:27:00.570460 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:00.570421    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:27:00.585473 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:00.585442    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:00.713126 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:00.713070    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hq8wj" podStartSLOduration=11.163130124 podStartE2EDuration="12.713049592s" totalImagesPullingTime="1.549919468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:51.686259497 +0000 UTC m=+41.857298537" watchObservedRunningTime="2026-07-28 01:27:00.713049592 +0000 UTC m=+50.884088629"
Jul 28 01:27:00.713295 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:00.713202    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cj2zl"]
Jul 28 01:27:00.716459 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:27:00.716434    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2960b4e1_5587_486f_95b3_aa18155fd6fe.slice/crio-d523c579e6d320091cf75d6be047bd798e1ab5a43923a7e2ae56f765f33328b6 WatchSource:0}: Error finding container d523c579e6d320091cf75d6be047bd798e1ab5a43923a7e2ae56f765f33328b6: Status 404 returned error can't find the container with id d523c579e6d320091cf75d6be047bd798e1ab5a43923a7e2ae56f765f33328b6
Jul 28 01:27:01.364006 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:01.363965    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:01.571317 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:01.571275    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:01.655908 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:01.655825    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cj2zl" event={"ID":"2960b4e1-5587-486f-95b3-aa18155fd6fe","Type":"ContainerStarted","Data":"d523c579e6d320091cf75d6be047bd798e1ab5a43923a7e2ae56f765f33328b6"}
Jul 28 01:27:02.071211 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:02.071179    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:02.165280 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:02.165243    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:02.561060 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:02.561027    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:03.130276 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:03.130186    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:27:03.132715 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:03.132684    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a78712d7-20fe-49b8-94c9-96ac63bd9015-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv8g\" (UID: \"a78712d7-20fe-49b8-94c9-96ac63bd9015\") " pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:27:03.177847 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:03.177818    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5vv8g"
Jul 28 01:27:03.183067 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:03.183038    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:03.289309 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:03.289285    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:03.329264 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:03.329234    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5vv8g"]
Jul 28 01:27:03.332286 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:27:03.332255    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78712d7_20fe_49b8_94c9_96ac63bd9015.slice/crio-fc0f1d4bda05938fa49cd092c7438b898089e1b17174e053c2556b34a7d382e4 WatchSource:0}: Error finding container fc0f1d4bda05938fa49cd092c7438b898089e1b17174e053c2556b34a7d382e4: Status 404 returned error can't find the container with id fc0f1d4bda05938fa49cd092c7438b898089e1b17174e053c2556b34a7d382e4
Jul 28 01:27:03.381707 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:03.381630    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:03.661864 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:03.661777    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vv8g" event={"ID":"a78712d7-20fe-49b8-94c9-96ac63bd9015","Type":"ContainerStarted","Data":"fc0f1d4bda05938fa49cd092c7438b898089e1b17174e053c2556b34a7d382e4"}
Jul 28 01:27:03.663565 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:03.663541    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cj2zl" event={"ID":"2960b4e1-5587-486f-95b3-aa18155fd6fe","Type":"ContainerStarted","Data":"5d6c51c69c178d30f1fea7730a5141c3b0f2d0e1480bce231ffe173a0351b441"}
Jul 28 01:27:03.663677 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:03.663590    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cj2zl" event={"ID":"2960b4e1-5587-486f-95b3-aa18155fd6fe","Type":"ContainerStarted","Data":"960b0f9e9e40461c575a3959c1551d6bfcf2f26688a7ab603221617dfda28fa3"}
Jul 28 01:27:03.663729 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:03.663692    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:27:03.681403 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:03.681349    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cj2zl" podStartSLOduration=17.812623794 podStartE2EDuration="19.68133661s" totalImagesPullingTime="1.868712816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:03.68044961 +0000 UTC m=+53.851488648" watchObservedRunningTime="2026-07-28 01:27:03.68133661 +0000 UTC m=+53.852375648"
Jul 28 01:27:03.773133 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:03.764302    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:03.964936 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:03.964855    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:04.168893 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:04.168824    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:04.579129 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:04.579097    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:04.772329 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:04.772285    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:08.680772 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:08.680740    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vv8g" event={"ID":"a78712d7-20fe-49b8-94c9-96ac63bd9015","Type":"ContainerStarted","Data":"b11ace31ccc9e681e981634e7cb5d50e24830fb38e0307d9a1a897a0a62b4355"}
Jul 28 01:27:10.604066 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:10.604035    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8dgm"
Jul 28 01:27:11.973708 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:11.973613    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:12.076413 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.076365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:12.173966 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.173931    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:12.273751 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.273711    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:12.447253 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.447213    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:12.631282 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.631203    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:12.695876 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.695833    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vv8g" event={"ID":"a78712d7-20fe-49b8-94c9-96ac63bd9015","Type":"ContainerStarted","Data":"5cbae8a6147a0a7c6c16ad261e644809e98bfa31f0071c340a90e6f6a944cbc0"}
Jul 28 01:27:12.719564 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.719529    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:12.876279 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.876244    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"]
Jul 28 01:27:12.878558 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.878534    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"
Jul 28 01:27:12.881847 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.881794    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 01:27:12.882050 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.882017    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 01:27:12.882764 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.882741    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 01:27:12.882764 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.882755    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 01:27:12.886369 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.886352    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"]
Jul 28 01:27:12.888474 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.888453    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:12.895359 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.895337    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"]
Jul 28 01:27:12.895551 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.895521    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 01:27:12.895723 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.895704    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 01:27:12.897518 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.897496    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 01:27:12.897627 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.897502    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 01:27:12.901221 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.901199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhbv9\" (UniqueName: \"kubernetes.io/projected/47cd87ee-5817-421f-9a7f-464e45d95199-kube-api-access-dhbv9\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:12.901357 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.901242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f448477f-04b1-4965-befd-a95eea618348-tmp\") pod \"klusterlet-addon-workmgr-585c64fbfd-sbmrq\" (UID: \"f448477f-04b1-4965-befd-a95eea618348\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"
Jul 28 01:27:12.901357 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.901267    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khd2m\" (UniqueName: \"kubernetes.io/projected/f448477f-04b1-4965-befd-a95eea618348-kube-api-access-khd2m\") pod \"klusterlet-addon-workmgr-585c64fbfd-sbmrq\" (UID: \"f448477f-04b1-4965-befd-a95eea618348\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"
Jul 28 01:27:12.901472 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.901353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cd87ee-5817-421f-9a7f-464e45d95199-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:12.901472 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.901451    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/47cd87ee-5817-421f-9a7f-464e45d95199-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:12.901550 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.901503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f448477f-04b1-4965-befd-a95eea618348-klusterlet-config\") pod \"klusterlet-addon-workmgr-585c64fbfd-sbmrq\" (UID: \"f448477f-04b1-4965-befd-a95eea618348\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"
Jul 28 01:27:12.901550 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.901528    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/47cd87ee-5817-421f-9a7f-464e45d95199-ca\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:12.901623 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.901560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/47cd87ee-5817-421f-9a7f-464e45d95199-hub\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:12.901623 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.901584    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47cd87ee-5817-421f-9a7f-464e45d95199-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:12.909331 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:12.909276    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"]
Jul 28 01:27:13.002284 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.002248    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/47cd87ee-5817-421f-9a7f-464e45d95199-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:13.002779 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.002323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f448477f-04b1-4965-befd-a95eea618348-klusterlet-config\") pod \"klusterlet-addon-workmgr-585c64fbfd-sbmrq\" (UID: \"f448477f-04b1-4965-befd-a95eea618348\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"
Jul 28 01:27:13.002779 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.002348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/47cd87ee-5817-421f-9a7f-464e45d95199-ca\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:13.002779 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.002377    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/47cd87ee-5817-421f-9a7f-464e45d95199-hub\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:13.002779 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.002418    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47cd87ee-5817-421f-9a7f-464e45d95199-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:13.002779 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.002466    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhbv9\" (UniqueName: \"kubernetes.io/projected/47cd87ee-5817-421f-9a7f-464e45d95199-kube-api-access-dhbv9\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:13.002779 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.002501    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f448477f-04b1-4965-befd-a95eea618348-tmp\") pod \"klusterlet-addon-workmgr-585c64fbfd-sbmrq\" (UID: \"f448477f-04b1-4965-befd-a95eea618348\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"
Jul 28 01:27:13.002779 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.002529    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khd2m\" (UniqueName: \"kubernetes.io/projected/f448477f-04b1-4965-befd-a95eea618348-kube-api-access-khd2m\") pod \"klusterlet-addon-workmgr-585c64fbfd-sbmrq\" (UID: \"f448477f-04b1-4965-befd-a95eea618348\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"
Jul 28 01:27:13.002779 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.002564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cd87ee-5817-421f-9a7f-464e45d95199-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:13.003294 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.003251    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cd87ee-5817-421f-9a7f-464e45d95199-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:13.003294 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.003282    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f448477f-04b1-4965-befd-a95eea618348-tmp\") pod \"klusterlet-addon-workmgr-585c64fbfd-sbmrq\" (UID: \"f448477f-04b1-4965-befd-a95eea618348\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"
Jul 28 01:27:13.005721 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.005697    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/47cd87ee-5817-421f-9a7f-464e45d95199-ca\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:13.005885 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.005860    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f448477f-04b1-4965-befd-a95eea618348-klusterlet-config\") pod \"klusterlet-addon-workmgr-585c64fbfd-sbmrq\" (UID: \"f448477f-04b1-4965-befd-a95eea618348\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"
Jul 28 01:27:13.005951 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.005932    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47cd87ee-5817-421f-9a7f-464e45d95199-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:13.006013 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.005943    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/47cd87ee-5817-421f-9a7f-464e45d95199-hub\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:13.006802 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.006784    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/47cd87ee-5817-421f-9a7f-464e45d95199-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:13.022041 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.022005    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhbv9\" (UniqueName: \"kubernetes.io/projected/47cd87ee-5817-421f-9a7f-464e45d95199-kube-api-access-dhbv9\") pod \"cluster-proxy-proxy-agent-85f6fdcfbb-blp75\" (UID: \"47cd87ee-5817-421f-9a7f-464e45d95199\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:13.023184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.023158    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:13.025096 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.025076    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khd2m\" (UniqueName: \"kubernetes.io/projected/f448477f-04b1-4965-befd-a95eea618348-kube-api-access-khd2m\") pod \"klusterlet-addon-workmgr-585c64fbfd-sbmrq\" (UID: \"f448477f-04b1-4965-befd-a95eea618348\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"
Jul 28 01:27:13.155137 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.155050    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:13.189621 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.189588    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"
Jul 28 01:27:13.209733 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.209707    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"
Jul 28 01:27:13.270419 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.270372    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:13.349039 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.349010    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"]
Jul 28 01:27:13.352266 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:27:13.352228    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf448477f_04b1_4965_befd_a95eea618348.slice/crio-e51a1bd55b28effad497c91b934e50196b1cd7d2d9cc3704d4b5af4952fe53ca WatchSource:0}: Error finding container e51a1bd55b28effad497c91b934e50196b1cd7d2d9cc3704d4b5af4952fe53ca: Status 404 returned error can't find the container with id e51a1bd55b28effad497c91b934e50196b1cd7d2d9cc3704d4b5af4952fe53ca
Jul 28 01:27:13.368311 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.368177    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75"]
Jul 28 01:27:13.371024 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:27:13.370998    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47cd87ee_5817_421f_9a7f_464e45d95199.slice/crio-ad33ed44880bbb68acd5b59cd7a236c243c30c2b62944f71c3ca8e38f6357e9c WatchSource:0}: Error finding container ad33ed44880bbb68acd5b59cd7a236c243c30c2b62944f71c3ca8e38f6357e9c: Status 404 returned error can't find the container with id ad33ed44880bbb68acd5b59cd7a236c243c30c2b62944f71c3ca8e38f6357e9c
Jul 28 01:27:13.668964 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.668932    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cj2zl"
Jul 28 01:27:13.704970 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.704928    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq" event={"ID":"f448477f-04b1-4965-befd-a95eea618348","Type":"ContainerStarted","Data":"e51a1bd55b28effad497c91b934e50196b1cd7d2d9cc3704d4b5af4952fe53ca"}
Jul 28 01:27:13.706271 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.706246    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75" event={"ID":"47cd87ee-5817-421f-9a7f-464e45d95199","Type":"ContainerStarted","Data":"ad33ed44880bbb68acd5b59cd7a236c243c30c2b62944f71c3ca8e38f6357e9c"}
Jul 28 01:27:13.856589 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.856549    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:13.924775 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:13.924696    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:14.047149 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:14.047117    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:14.743001 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:14.742942    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:15.716887 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:15.716852    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vv8g" event={"ID":"a78712d7-20fe-49b8-94c9-96ac63bd9015","Type":"ContainerStarted","Data":"b8266f99a3fc138bb4808ba8b54e0c0b47fff3c646e433731021a7a3974ba3a2"}
Jul 28 01:27:16.029567 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.029001    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:27:16.029567 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.029117    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs\") pod \"network-metrics-daemon-22fhh\" (UID: \"8849435f-9f9d-4fd2-858c-b037077d839c\") " pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:27:16.032244 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.031811    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 01:27:16.032244 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.032063    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 01:27:16.049195 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.049095    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8849435f-9f9d-4fd2-858c-b037077d839c-metrics-certs\") pod \"network-metrics-daemon-22fhh\" (UID: \"8849435f-9f9d-4fd2-858c-b037077d839c\") " pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:27:16.051181 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.051155    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be103d49-24c3-4a2e-9a55-2fcc960c1a71-original-pull-secret\") pod \"global-pull-secret-syncer-nf7cb\" (UID: \"be103d49-24c3-4a2e-9a55-2fcc960c1a71\") " pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:27:16.130558 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.130449    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqkct\" (UniqueName: \"kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct\") pod \"network-check-target-km9pg\" (UID: \"14745d0d-a327-4289-84d7-213c1e2f3eef\") " pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:27:16.133255 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.133052    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 01:27:16.138134 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.138109    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nf7cb"
Jul 28 01:27:16.143737 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.143518    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 01:27:16.153411 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.153367    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cq5js\""
Jul 28 01:27:16.162249 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.161667    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22fhh"
Jul 28 01:27:16.162698 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.162676    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqkct\" (UniqueName: \"kubernetes.io/projected/14745d0d-a327-4289-84d7-213c1e2f3eef-kube-api-access-hqkct\") pod \"network-check-target-km9pg\" (UID: \"14745d0d-a327-4289-84d7-213c1e2f3eef\") " pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:27:16.344713 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.344665    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5vv8g" podStartSLOduration=17.614182063 podStartE2EDuration="29.34464526s" totalImagesPullingTime="11.730463197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:15.740903668 +0000 UTC m=+65.911942709" watchObservedRunningTime="2026-07-28 01:27:16.34464526 +0000 UTC m=+66.515684299"
Jul 28 01:27:16.345125 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.345102    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nf7cb"]
Jul 28 01:27:16.358808 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.358775    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-22fhh"]
Jul 28 01:27:16.447680 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.447636    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lkkxh\""
Jul 28 01:27:16.455482 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.455452    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:27:16.685459 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.685366    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:16.879268 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.879235    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:16.976665 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:16.976557    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:17.078823 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:17.078777    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:17.216808 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:17.216778    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:17.314290 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:27:17.314259    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8849435f_9f9d_4fd2_858c_b037077d839c.slice/crio-a340927c42abc316e175a95d3a55d7f220eafff4d8e9a5c8ec8c4b8b5501452c WatchSource:0}: Error finding container a340927c42abc316e175a95d3a55d7f220eafff4d8e9a5c8ec8c4b8b5501452c: Status 404 returned error can't find the container with id a340927c42abc316e175a95d3a55d7f220eafff4d8e9a5c8ec8c4b8b5501452c
Jul 28 01:27:17.317365 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:27:17.316652    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe103d49_24c3_4a2e_9a55_2fcc960c1a71.slice/crio-92c74bf4da158bac8ef2c0ea2ea34304114b101c97af0aeb115c10bbcb3d578d WatchSource:0}: Error finding container 92c74bf4da158bac8ef2c0ea2ea34304114b101c97af0aeb115c10bbcb3d578d: Status 404 returned error can't find the container with id 92c74bf4da158bac8ef2c0ea2ea34304114b101c97af0aeb115c10bbcb3d578d
Jul 28 01:27:17.330310 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:17.330281    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:17.470493 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:17.470460    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-km9pg"]
Jul 28 01:27:17.474107 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:27:17.474069    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14745d0d_a327_4289_84d7_213c1e2f3eef.slice/crio-4718174aeba220e6a761967b7dab41433978ad4d0a8a194162222d19ad273cd9 WatchSource:0}: Error finding container 4718174aeba220e6a761967b7dab41433978ad4d0a8a194162222d19ad273cd9: Status 404 returned error can't find the container with id 4718174aeba220e6a761967b7dab41433978ad4d0a8a194162222d19ad273cd9
Jul 28 01:27:17.725960 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:17.725789    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-km9pg" event={"ID":"14745d0d-a327-4289-84d7-213c1e2f3eef","Type":"ContainerStarted","Data":"4718174aeba220e6a761967b7dab41433978ad4d0a8a194162222d19ad273cd9"}
Jul 28 01:27:17.727188 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:17.727157    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-22fhh" event={"ID":"8849435f-9f9d-4fd2-858c-b037077d839c","Type":"ContainerStarted","Data":"a340927c42abc316e175a95d3a55d7f220eafff4d8e9a5c8ec8c4b8b5501452c"}
Jul 28 01:27:17.728396 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:17.728358    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nf7cb" event={"ID":"be103d49-24c3-4a2e-9a55-2fcc960c1a71","Type":"ContainerStarted","Data":"92c74bf4da158bac8ef2c0ea2ea34304114b101c97af0aeb115c10bbcb3d578d"}
Jul 28 01:27:18.161836 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:18.161774    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:18.229708 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:18.229677    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:19.729903 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:19.729873    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:19.797599 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:19.797554    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:20.746960 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:20.746521    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq" event={"ID":"f448477f-04b1-4965-befd-a95eea618348","Type":"ContainerStarted","Data":"faeb3316882ac148d8e0cc4c05dfd05a3c49535f4b62a67e499196bf9234ac68"}
Jul 28 01:27:20.746960 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:20.746741    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"
Jul 28 01:27:20.748454 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:20.748356    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-22fhh" event={"ID":"8849435f-9f9d-4fd2-858c-b037077d839c","Type":"ContainerStarted","Data":"d0b7702f3e07b75d758a96a0b36835b6fb9930d3093873f533c80fbb6f63fbe8"}
Jul 28 01:27:20.748573 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:20.748534    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq"
Jul 28 01:27:20.750410 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:20.750367    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75" event={"ID":"47cd87ee-5817-421f-9a7f-464e45d95199","Type":"ContainerStarted","Data":"9cf9d3dddbc30d3847943ed6ceb745875ba2c4ee9e5ac29bbd572b41acb24dfd"}
Jul 28 01:27:20.761994 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:20.761951    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585c64fbfd-sbmrq" podStartSLOduration=1.587530114 podStartE2EDuration="8.761938638s" totalImagesPullingTime="7.174408524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:20.761152179 +0000 UTC m=+70.932191216" watchObservedRunningTime="2026-07-28 01:27:20.761938638 +0000 UTC m=+70.932977676"
Jul 28 01:27:22.016393 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:22.016356    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:22.121672 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:22.121557    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:22.183683 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:22.183648    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:22.339934 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:22.339893    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:22.421545 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:22.421470    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:22.759734 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:22.759698    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-km9pg" event={"ID":"14745d0d-a327-4289-84d7-213c1e2f3eef","Type":"ContainerStarted","Data":"dca2c379fa143e36ddda5023a6f4393b5dda53e65b3c98f588be206f309c174e"}
Jul 28 01:27:22.759929 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:22.759767    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:27:22.761788 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:22.761757    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-22fhh" event={"ID":"8849435f-9f9d-4fd2-858c-b037077d839c","Type":"ContainerStarted","Data":"b7294f249107720ee26e811f86f38f2bd2e351d97d53144399d53b06f6fd6696"}
Jul 28 01:27:22.765621 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:22.765592    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75" event={"ID":"47cd87ee-5817-421f-9a7f-464e45d95199","Type":"ContainerStarted","Data":"02dbd77375858fb4a28ffd1e459d7bddccd6c50ed3a60838de2533f634f54f3f"}
Jul 28 01:27:22.765727 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:22.765635    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75" event={"ID":"47cd87ee-5817-421f-9a7f-464e45d95199","Type":"ContainerStarted","Data":"2b2e476c04986a611028dd1f4190b0f0ac14d3260bddb9fb4653d8a98028a722"}
Jul 28 01:27:22.780556 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:22.780503    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-km9pg" podStartSLOduration=68.394424383 podStartE2EDuration="1m12.780484419s" totalImagesPullingTime="4.386060036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:22.780080601 +0000 UTC m=+72.951119658" watchObservedRunningTime="2026-07-28 01:27:22.780484419 +0000 UTC m=+72.951523456"
Jul 28 01:27:22.807832 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:22.807763    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-22fhh" podStartSLOduration=69.555523966 podStartE2EDuration="1m12.807745784s" totalImagesPullingTime="3.252221818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:22.807496252 +0000 UTC m=+72.978535316" watchObservedRunningTime="2026-07-28 01:27:22.807745784 +0000 UTC m=+72.978784827"
Jul 28 01:27:22.834634 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:22.834576    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f6fdcfbb-blp75" podStartSLOduration=3.678812636 podStartE2EDuration="10.834554275s" totalImagesPullingTime="7.155741639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:22.833558617 +0000 UTC m=+73.004597655" watchObservedRunningTime="2026-07-28 01:27:22.834554275 +0000 UTC m=+73.005593313"
Jul 28 01:27:22.953530 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:22.953497    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:23.043718 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:23.043637    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:23.146820 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:23.146788    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:24.773563 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:24.773526    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nf7cb" event={"ID":"be103d49-24c3-4a2e-9a55-2fcc960c1a71","Type":"ContainerStarted","Data":"4870d9bd25f7838b2b6ad529764abc8202b0e0372764cad08c49a8e2ac917c43"}
Jul 28 01:27:25.076749 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:25.076673    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:25.150090 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:25.150058    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:25.237047 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:25.237006    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:25.336528 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:25.336430    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:27.609049 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:27.609013    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:28.844634 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:28.844596    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:28.939560 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:28.939522    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:29.090004 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:29.089972    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:29.179051 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:29.178969    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:31.673161 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:31.673049    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:31.772949 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:31.772909    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:33.146128 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:33.146069    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:33.231931 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:33.231886    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:34.693225 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:34.693172    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:34.798115 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:34.798074    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:35.416960 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:35.416926    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:37.307934 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.307902    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:37.533085 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.533053    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:37.647752 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.647646    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nf7cb" podStartSLOduration=81.17590662 podStartE2EDuration="1m27.647630242s" totalImagesPullingTime="6.471723622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:24.794841497 +0000 UTC m=+74.965880534" watchObservedRunningTime="2026-07-28 01:27:37.647630242 +0000 UTC m=+87.818669294"
Jul 28 01:27:37.648128 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.648112    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cngcv"]
Jul 28 01:27:37.653702 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.653675    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.656166 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.656134    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 01:27:37.656473 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.656445    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 01:27:37.656553 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.656490    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 01:27:37.657151 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.657135    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 01:27:37.657205 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.657162    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 01:27:37.657248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.657176    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 01:27:37.657248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.657174    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s548w\""
Jul 28 01:27:37.682129 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.682105    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:37.766868 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.766833    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:37.781414 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.781374    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-node-exporter-wtmp\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.781495 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.781420    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.781550 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.781490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-metrics-client-ca\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.781550 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.781517    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-node-exporter-textfile\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.781642 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.781548    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-sys\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.781642 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.781587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-node-exporter-accelerators-collector-config\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.781642 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.781634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6wdk\" (UniqueName: \"kubernetes.io/projected/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-kube-api-access-f6wdk\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.781770 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.781677    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-root\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.781770 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.781706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-node-exporter-tls\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.867218 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.867183    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:37.882454 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.882424    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-node-exporter-wtmp\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.882550 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.882465    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.882550 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.882510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-metrics-client-ca\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.882550 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.882536    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-node-exporter-textfile\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.882662 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.882560    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-sys\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.882662 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.882603    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-node-exporter-wtmp\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.882662 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.882624    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-sys\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.882662 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.882646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-node-exporter-accelerators-collector-config\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.882835 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.882672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6wdk\" (UniqueName: \"kubernetes.io/projected/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-kube-api-access-f6wdk\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.882835 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.882697    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-root\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.882835 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.882723    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-node-exporter-tls\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.882835 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.882778    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-root\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.883014 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.882987    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-node-exporter-textfile\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.883221 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.883203    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-metrics-client-ca\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.883297 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.883270    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-node-exporter-accelerators-collector-config\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.884996 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.884976    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.885159 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.885142    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-node-exporter-tls\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.893111 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.893092    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6wdk\" (UniqueName: \"kubernetes.io/projected/297c5f42-e728-4ee6-92c9-ac2cefeb2fa0-kube-api-access-f6wdk\") pod \"node-exporter-cngcv\" (UID: \"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0\") " pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.962280 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.962157    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:37.963082 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:37.963065    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cngcv"
Jul 28 01:27:37.976058 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:27:37.976029    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297c5f42_e728_4ee6_92c9_ac2cefeb2fa0.slice/crio-58cb8c6ff993c977d6dfbfa0b03a106f1e792e7cf40d43f54a03cbd2599745b2 WatchSource:0}: Error finding container 58cb8c6ff993c977d6dfbfa0b03a106f1e792e7cf40d43f54a03cbd2599745b2: Status 404 returned error can't find the container with id 58cb8c6ff993c977d6dfbfa0b03a106f1e792e7cf40d43f54a03cbd2599745b2
Jul 28 01:27:38.039565 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:38.039531    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:38.140905 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:38.140830    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:38.305051 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:38.305023    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:38.444261 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:38.444227    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:38.544062 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:38.544036    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:38.630721 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:38.630642    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:38.758729 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:38.758690    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:38.825992 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:38.825963    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cngcv" event={"ID":"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0","Type":"ContainerStarted","Data":"58cb8c6ff993c977d6dfbfa0b03a106f1e792e7cf40d43f54a03cbd2599745b2"}
Jul 28 01:27:38.959911 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:38.959880    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:39.831211 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:39.831176    2576 generic.go:356] "Generic (PLEG): container finished" podID="297c5f42-e728-4ee6-92c9-ac2cefeb2fa0" containerID="2ba3628fd5959ace345c2ac57963bd00bdc15ed5176edd5c531ee6f4b14e6f35" exitCode=0
Jul 28 01:27:39.831645 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:39.831223    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cngcv" event={"ID":"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0","Type":"ContainerDied","Data":"2ba3628fd5959ace345c2ac57963bd00bdc15ed5176edd5c531ee6f4b14e6f35"}
Jul 28 01:27:40.026550 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:40.026517    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:40.148007 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:40.147932    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:40.248851 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:40.248819    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:40.550667 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:40.550635    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:40.650246 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:40.650212    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:40.742326 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:40.742290    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:40.836887 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:40.836806    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cngcv" event={"ID":"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0","Type":"ContainerStarted","Data":"ad47d38e0b100f4ec817b2640b6490666f63a87b4299c6e954ad35708c140337"}
Jul 28 01:27:40.836887 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:40.836868    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cngcv" event={"ID":"297c5f42-e728-4ee6-92c9-ac2cefeb2fa0","Type":"ContainerStarted","Data":"9f62b711793397a8fbfd24f21b74c3ffdfe1e110945840e5a60d75db8186a890"}
Jul 28 01:27:40.887339 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:40.887292    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cngcv" podStartSLOduration=3.016513444 podStartE2EDuration="3.887275914s" totalImagesPullingTime="798.735138ms" totalInitContainerRuntime="72.027332ms" isStatefulPod=true podCreationTimestamp="2026-07-28 01:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:40.885940765 +0000 UTC m=+91.056979801" watchObservedRunningTime="2026-07-28 01:27:40.887275914 +0000 UTC m=+91.058314952"
Jul 28 01:27:41.330796 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:41.330763    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:42.323286 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:42.323256    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:42.448323 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:42.448294    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:43.594553 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:43.594521    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:43.841626 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:43.841590    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:44.758165 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:44.758129    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:44.850159 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:44.850122    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:48.148712 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:48.148670    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:48.214852 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:48.214815    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:53.772057 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:53.772026    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-km9pg"
Jul 28 01:27:58.073582 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:58.073539    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:58.173538 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:58.173504    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:59.477762 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:59.477729    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:59.540709 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:27:59.540677    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:03.144298 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:28:03.144260    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:03.211654 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:28:03.211621    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:03.875326 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:28:03.875289    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:03.976511 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:28:03.976472    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:04.051216 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:28:04.051163    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:11.927275 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:28:11.927239    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:12.028147 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:28:12.028113    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:12.086401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:28:12.086352    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:18.149970 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:28:18.149935    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:18.211254 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:28:18.211227    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:53.723660 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:28:53.723624    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:53.789047 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:28:53.789010    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:01.662684 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:29:01.662596    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:01.763633 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:29:01.763603    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:05.668674 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:29:05.668638    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:05.770007 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:29:05.769972    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:12.175649 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:29:12.175611    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:12.234747 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:29:12.234718    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:18.151580 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:29:18.151546    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:18.252126 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:29:18.252041    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:28.175098 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:29:28.175024    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:00.406194 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:30:00.406150    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:07.780542 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:30:07.780475    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:07.835647 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:30:07.835617    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:37.008800 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:30:37.008696    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:37.067044 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:30:37.066995    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:10.237875 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:10.237848    2576 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 01:31:12.733728 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:12.733656    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:15.428953 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:15.428916    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:35.617154 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:35.617111    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:35.712998 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:35.712959    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:41.967418 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:41.967369    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:42.248524 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:42.248492    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:42.460524 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:42.460484    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:42.559338 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:42.559253    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:42.977324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:42.977241    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:43.074740 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:43.074706    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:43.177100 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:43.177067    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:43.277893 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:43.277863    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:43.374011 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:43.373961    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:43.521612 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:43.521569    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:43.683054 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:43.682969    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:43.782293 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:31:43.782262    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:05.591027 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:32:05.590942    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:57.840044 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:32:57.840002    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:59.919328 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:32:59.919298    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:59.991930 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:32:59.991898    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-xvddh"]
Jul 28 01:32:59.993916 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:32:59.993899    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-xvddh"
Jul 28 01:32:59.995690 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:32:59.995667    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:59.996552 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:32:59.996540    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 01:32:59.997307 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:32:59.997293    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-rgtq8\""
Jul 28 01:32:59.997612 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:32:59.997596    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 01:32:59.998462 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:32:59.998448    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 01:33:00.010186 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.010165    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-tmxdx"]
Jul 28 01:33:00.012035 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.012017    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-tmxdx"
Jul 28 01:33:00.014669 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.014648    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 01:33:00.014816 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.014720    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-k78hw\""
Jul 28 01:33:00.015046 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.015023    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-xvddh"]
Jul 28 01:33:00.029905 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.029881    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-tmxdx"]
Jul 28 01:33:00.094513 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.094482    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88705942-2637-4b34-a67a-3cc1b9c5442a-cert\") pod \"kserve-controller-manager-ff96d5959-xvddh\" (UID: \"88705942-2637-4b34-a67a-3cc1b9c5442a\") " pod="kserve/kserve-controller-manager-ff96d5959-xvddh"
Jul 28 01:33:00.094674 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.094521    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg82w\" (UniqueName: \"kubernetes.io/projected/88705942-2637-4b34-a67a-3cc1b9c5442a-kube-api-access-kg82w\") pod \"kserve-controller-manager-ff96d5959-xvddh\" (UID: \"88705942-2637-4b34-a67a-3cc1b9c5442a\") " pod="kserve/kserve-controller-manager-ff96d5959-xvddh"
Jul 28 01:33:00.112209 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.112178    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:00.195059 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.194977    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adc41827-89af-4789-81c2-30c4f420f89d-cert\") pod \"llmisvc-controller-manager-84fc959947-tmxdx\" (UID: \"adc41827-89af-4789-81c2-30c4f420f89d\") " pod="kserve/llmisvc-controller-manager-84fc959947-tmxdx"
Jul 28 01:33:00.195059 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.195018    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59snx\" (UniqueName: \"kubernetes.io/projected/adc41827-89af-4789-81c2-30c4f420f89d-kube-api-access-59snx\") pod \"llmisvc-controller-manager-84fc959947-tmxdx\" (UID: \"adc41827-89af-4789-81c2-30c4f420f89d\") " pod="kserve/llmisvc-controller-manager-84fc959947-tmxdx"
Jul 28 01:33:00.195226 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.195074    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88705942-2637-4b34-a67a-3cc1b9c5442a-cert\") pod \"kserve-controller-manager-ff96d5959-xvddh\" (UID: \"88705942-2637-4b34-a67a-3cc1b9c5442a\") " pod="kserve/kserve-controller-manager-ff96d5959-xvddh"
Jul 28 01:33:00.195226 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.195162    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg82w\" (UniqueName: \"kubernetes.io/projected/88705942-2637-4b34-a67a-3cc1b9c5442a-kube-api-access-kg82w\") pod \"kserve-controller-manager-ff96d5959-xvddh\" (UID: \"88705942-2637-4b34-a67a-3cc1b9c5442a\") " pod="kserve/kserve-controller-manager-ff96d5959-xvddh"
Jul 28 01:33:00.197719 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.197689    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88705942-2637-4b34-a67a-3cc1b9c5442a-cert\") pod \"kserve-controller-manager-ff96d5959-xvddh\" (UID: \"88705942-2637-4b34-a67a-3cc1b9c5442a\") " pod="kserve/kserve-controller-manager-ff96d5959-xvddh"
Jul 28 01:33:00.203653 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.203632    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg82w\" (UniqueName: \"kubernetes.io/projected/88705942-2637-4b34-a67a-3cc1b9c5442a-kube-api-access-kg82w\") pod \"kserve-controller-manager-ff96d5959-xvddh\" (UID: \"88705942-2637-4b34-a67a-3cc1b9c5442a\") " pod="kserve/kserve-controller-manager-ff96d5959-xvddh"
Jul 28 01:33:00.207017 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.206997    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:00.296115 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.296087    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59snx\" (UniqueName: \"kubernetes.io/projected/adc41827-89af-4789-81c2-30c4f420f89d-kube-api-access-59snx\") pod \"llmisvc-controller-manager-84fc959947-tmxdx\" (UID: \"adc41827-89af-4789-81c2-30c4f420f89d\") " pod="kserve/llmisvc-controller-manager-84fc959947-tmxdx"
Jul 28 01:33:00.296260 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.296137    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adc41827-89af-4789-81c2-30c4f420f89d-cert\") pod \"llmisvc-controller-manager-84fc959947-tmxdx\" (UID: \"adc41827-89af-4789-81c2-30c4f420f89d\") " pod="kserve/llmisvc-controller-manager-84fc959947-tmxdx"
Jul 28 01:33:00.298710 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.298689    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adc41827-89af-4789-81c2-30c4f420f89d-cert\") pod \"llmisvc-controller-manager-84fc959947-tmxdx\" (UID: \"adc41827-89af-4789-81c2-30c4f420f89d\") " pod="kserve/llmisvc-controller-manager-84fc959947-tmxdx"
Jul 28 01:33:00.303791 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.303770    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-xvddh"
Jul 28 01:33:00.306153 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.305802    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59snx\" (UniqueName: \"kubernetes.io/projected/adc41827-89af-4789-81c2-30c4f420f89d-kube-api-access-59snx\") pod \"llmisvc-controller-manager-84fc959947-tmxdx\" (UID: \"adc41827-89af-4789-81c2-30c4f420f89d\") " pod="kserve/llmisvc-controller-manager-84fc959947-tmxdx"
Jul 28 01:33:00.322978 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.322952    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-tmxdx"
Jul 28 01:33:00.452934 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.452865    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-xvddh"]
Jul 28 01:33:00.456321 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:33:00.456302    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88705942_2637_4b34_a67a_3cc1b9c5442a.slice/crio-cc57e05aebba0d82e68379813e2a3c9dcf6bc8d3c1e7c1a8b5c35a7b426b162f WatchSource:0}: Error finding container cc57e05aebba0d82e68379813e2a3c9dcf6bc8d3c1e7c1a8b5c35a7b426b162f: Status 404 returned error can't find the container with id cc57e05aebba0d82e68379813e2a3c9dcf6bc8d3c1e7c1a8b5c35a7b426b162f
Jul 28 01:33:00.457519 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.457500    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:33:00.480376 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.480353    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-tmxdx"]
Jul 28 01:33:00.482920 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:33:00.482897    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podadc41827_89af_4789_81c2_30c4f420f89d.slice/crio-854a2865ad753aa498ff29e50b2153d0920592fb12d320dc8cb572f9047c6cf7 WatchSource:0}: Error finding container 854a2865ad753aa498ff29e50b2153d0920592fb12d320dc8cb572f9047c6cf7: Status 404 returned error can't find the container with id 854a2865ad753aa498ff29e50b2153d0920592fb12d320dc8cb572f9047c6cf7
Jul 28 01:33:00.847440 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.847396    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-xvddh" event={"ID":"88705942-2637-4b34-a67a-3cc1b9c5442a","Type":"ContainerStarted","Data":"cc57e05aebba0d82e68379813e2a3c9dcf6bc8d3c1e7c1a8b5c35a7b426b162f"}
Jul 28 01:33:00.848540 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.848499    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-tmxdx" event={"ID":"adc41827-89af-4789-81c2-30c4f420f89d","Type":"ContainerStarted","Data":"854a2865ad753aa498ff29e50b2153d0920592fb12d320dc8cb572f9047c6cf7"}
Jul 28 01:33:00.959334 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:00.959298    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:01.074153 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:01.074114    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:03.860099 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:03.860066    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-xvddh" event={"ID":"88705942-2637-4b34-a67a-3cc1b9c5442a","Type":"ContainerStarted","Data":"35ac83fcd60ac4c35bf119bc2c67dde4448d240a2a856485e87358b9a368b5cd"}
Jul 28 01:33:03.860526 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:03.860221    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-xvddh"
Jul 28 01:33:03.879721 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:03.879671    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-xvddh" podStartSLOduration=2.441949313 podStartE2EDuration="4.879646615s" totalImagesPullingTime="2.437697302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:33:03.877998988 +0000 UTC m=+414.049038026" watchObservedRunningTime="2026-07-28 01:33:03.879646615 +0000 UTC m=+414.050685652"
Jul 28 01:33:05.868335 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:05.868300    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-tmxdx" event={"ID":"adc41827-89af-4789-81c2-30c4f420f89d","Type":"ContainerStarted","Data":"558a927409ab8c1f2e5caf339b3603ba07d3d71524413afa9034d66023531007"}
Jul 28 01:33:05.868767 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:05.868437    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84fc959947-tmxdx"
Jul 28 01:33:05.885272 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:05.885229    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84fc959947-tmxdx" podStartSLOduration=2.5351891589999997 podStartE2EDuration="6.885215102s" totalImagesPullingTime="4.350025943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:33:05.884428482 +0000 UTC m=+416.055467518" watchObservedRunningTime="2026-07-28 01:33:05.885215102 +0000 UTC m=+416.056254139"
Jul 28 01:33:09.231951 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:09.231916    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:09.331551 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:09.331515    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:09.399439 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:09.399404    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:12.228952 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:12.228912    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:12.304433 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:12.304402    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:34.869184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:34.869110    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-xvddh"
Jul 28 01:33:36.874257 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:36.874223    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84fc959947-tmxdx"
Jul 28 01:33:45.071767 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.071729    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-xvddh"]
Jul 28 01:33:45.072350 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.071992    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-ff96d5959-xvddh" podUID="88705942-2637-4b34-a67a-3cc1b9c5442a" containerName="manager" containerID="cri-o://35ac83fcd60ac4c35bf119bc2c67dde4448d240a2a856485e87358b9a368b5cd" gracePeriod=10
Jul 28 01:33:45.107140 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.107108    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-nzsqn"]
Jul 28 01:33:45.109472 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.109455    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-nzsqn"
Jul 28 01:33:45.132894 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.132865    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-nzsqn"]
Jul 28 01:33:45.198542 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.198516    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n27mc\" (UniqueName: \"kubernetes.io/projected/be0d01d3-4102-4a71-81eb-30fc23f7a68f-kube-api-access-n27mc\") pod \"kserve-controller-manager-ff96d5959-nzsqn\" (UID: \"be0d01d3-4102-4a71-81eb-30fc23f7a68f\") " pod="kserve/kserve-controller-manager-ff96d5959-nzsqn"
Jul 28 01:33:45.198641 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.198563    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be0d01d3-4102-4a71-81eb-30fc23f7a68f-cert\") pod \"kserve-controller-manager-ff96d5959-nzsqn\" (UID: \"be0d01d3-4102-4a71-81eb-30fc23f7a68f\") " pod="kserve/kserve-controller-manager-ff96d5959-nzsqn"
Jul 28 01:33:45.299683 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.299655    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be0d01d3-4102-4a71-81eb-30fc23f7a68f-cert\") pod \"kserve-controller-manager-ff96d5959-nzsqn\" (UID: \"be0d01d3-4102-4a71-81eb-30fc23f7a68f\") " pod="kserve/kserve-controller-manager-ff96d5959-nzsqn"
Jul 28 01:33:45.299801 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.299728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27mc\" (UniqueName: \"kubernetes.io/projected/be0d01d3-4102-4a71-81eb-30fc23f7a68f-kube-api-access-n27mc\") pod \"kserve-controller-manager-ff96d5959-nzsqn\" (UID: \"be0d01d3-4102-4a71-81eb-30fc23f7a68f\") " pod="kserve/kserve-controller-manager-ff96d5959-nzsqn"
Jul 28 01:33:45.302089 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.302066    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be0d01d3-4102-4a71-81eb-30fc23f7a68f-cert\") pod \"kserve-controller-manager-ff96d5959-nzsqn\" (UID: \"be0d01d3-4102-4a71-81eb-30fc23f7a68f\") " pod="kserve/kserve-controller-manager-ff96d5959-nzsqn"
Jul 28 01:33:45.308944 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.308915    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27mc\" (UniqueName: \"kubernetes.io/projected/be0d01d3-4102-4a71-81eb-30fc23f7a68f-kube-api-access-n27mc\") pod \"kserve-controller-manager-ff96d5959-nzsqn\" (UID: \"be0d01d3-4102-4a71-81eb-30fc23f7a68f\") " pod="kserve/kserve-controller-manager-ff96d5959-nzsqn"
Jul 28 01:33:45.312253 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.312234    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-xvddh"
Jul 28 01:33:45.400747 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.400667    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88705942-2637-4b34-a67a-3cc1b9c5442a-cert\") pod \"88705942-2637-4b34-a67a-3cc1b9c5442a\" (UID: \"88705942-2637-4b34-a67a-3cc1b9c5442a\") "
Jul 28 01:33:45.400747 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.400743    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg82w\" (UniqueName: \"kubernetes.io/projected/88705942-2637-4b34-a67a-3cc1b9c5442a-kube-api-access-kg82w\") pod \"88705942-2637-4b34-a67a-3cc1b9c5442a\" (UID: \"88705942-2637-4b34-a67a-3cc1b9c5442a\") "
Jul 28 01:33:45.402991 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.402960    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88705942-2637-4b34-a67a-3cc1b9c5442a-kube-api-access-kg82w" (OuterVolumeSpecName: "kube-api-access-kg82w") pod "88705942-2637-4b34-a67a-3cc1b9c5442a" (UID: "88705942-2637-4b34-a67a-3cc1b9c5442a"). InnerVolumeSpecName "kube-api-access-kg82w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:33:45.403111 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.402964    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88705942-2637-4b34-a67a-3cc1b9c5442a-cert" (OuterVolumeSpecName: "cert") pod "88705942-2637-4b34-a67a-3cc1b9c5442a" (UID: "88705942-2637-4b34-a67a-3cc1b9c5442a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:33:45.455350 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.455303    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-nzsqn"
Jul 28 01:33:45.502104 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.502067    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg82w\" (UniqueName: \"kubernetes.io/projected/88705942-2637-4b34-a67a-3cc1b9c5442a-kube-api-access-kg82w\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:33:45.502213 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.502110    2576 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88705942-2637-4b34-a67a-3cc1b9c5442a-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:33:45.579856 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.579830    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-nzsqn"]
Jul 28 01:33:45.584780 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:33:45.584757    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0d01d3_4102_4a71_81eb_30fc23f7a68f.slice/crio-b0276274e7d171beffbefab5a9bdf9f62e2efc5ee1df3968f6df3b9d921a705f WatchSource:0}: Error finding container b0276274e7d171beffbefab5a9bdf9f62e2efc5ee1df3968f6df3b9d921a705f: Status 404 returned error can't find the container with id b0276274e7d171beffbefab5a9bdf9f62e2efc5ee1df3968f6df3b9d921a705f
Jul 28 01:33:45.997924 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.997894    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-nzsqn" event={"ID":"be0d01d3-4102-4a71-81eb-30fc23f7a68f","Type":"ContainerStarted","Data":"b0276274e7d171beffbefab5a9bdf9f62e2efc5ee1df3968f6df3b9d921a705f"}
Jul 28 01:33:45.999023 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.999002    2576 generic.go:356] "Generic (PLEG): container finished" podID="88705942-2637-4b34-a67a-3cc1b9c5442a" containerID="35ac83fcd60ac4c35bf119bc2c67dde4448d240a2a856485e87358b9a368b5cd" exitCode=0
Jul 28 01:33:45.999117 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.999030    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-xvddh" event={"ID":"88705942-2637-4b34-a67a-3cc1b9c5442a","Type":"ContainerDied","Data":"35ac83fcd60ac4c35bf119bc2c67dde4448d240a2a856485e87358b9a368b5cd"}
Jul 28 01:33:45.999117 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.999044    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-xvddh"
Jul 28 01:33:45.999117 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.999058    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-xvddh" event={"ID":"88705942-2637-4b34-a67a-3cc1b9c5442a","Type":"ContainerDied","Data":"cc57e05aebba0d82e68379813e2a3c9dcf6bc8d3c1e7c1a8b5c35a7b426b162f"}
Jul 28 01:33:45.999117 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:45.999078    2576 scope.go:117] "RemoveContainer" containerID="35ac83fcd60ac4c35bf119bc2c67dde4448d240a2a856485e87358b9a368b5cd"
Jul 28 01:33:46.009995 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:46.009975    2576 scope.go:117] "RemoveContainer" containerID="35ac83fcd60ac4c35bf119bc2c67dde4448d240a2a856485e87358b9a368b5cd"
Jul 28 01:33:46.010260 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:33:46.010234    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ac83fcd60ac4c35bf119bc2c67dde4448d240a2a856485e87358b9a368b5cd\": container with ID starting with 35ac83fcd60ac4c35bf119bc2c67dde4448d240a2a856485e87358b9a368b5cd not found: ID does not exist" containerID="35ac83fcd60ac4c35bf119bc2c67dde4448d240a2a856485e87358b9a368b5cd"
Jul 28 01:33:46.010343 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:46.010269    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ac83fcd60ac4c35bf119bc2c67dde4448d240a2a856485e87358b9a368b5cd"} err="failed to get container status \"35ac83fcd60ac4c35bf119bc2c67dde4448d240a2a856485e87358b9a368b5cd\": rpc error: code = NotFound desc = could not find container \"35ac83fcd60ac4c35bf119bc2c67dde4448d240a2a856485e87358b9a368b5cd\": container with ID starting with 35ac83fcd60ac4c35bf119bc2c67dde4448d240a2a856485e87358b9a368b5cd not found: ID does not exist"
Jul 28 01:33:46.019654 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:46.019630    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-xvddh"]
Jul 28 01:33:46.023185 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:46.023167    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-xvddh"]
Jul 28 01:33:46.431029 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:46.430997    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88705942-2637-4b34-a67a-3cc1b9c5442a" path="/var/lib/kubelet/pods/88705942-2637-4b34-a67a-3cc1b9c5442a/volumes"
Jul 28 01:33:47.004233 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:47.004198    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-nzsqn" event={"ID":"be0d01d3-4102-4a71-81eb-30fc23f7a68f","Type":"ContainerStarted","Data":"a42c151c71ce23a294932170a3766749b1bcd5beab831d96d3b172cf499cba4d"}
Jul 28 01:33:47.004463 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:47.004331    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-nzsqn"
Jul 28 01:33:47.025337 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:33:47.025293    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-nzsqn" podStartSLOduration=1.639620681 podStartE2EDuration="2.025280653s" totalImagesPullingTime="385.659972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:33:47.023106632 +0000 UTC m=+457.194145669" watchObservedRunningTime="2026-07-28 01:33:47.025280653 +0000 UTC m=+457.196319689"
Jul 28 01:34:16.030791 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:34:16.030755    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:18.013513 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:34:18.013481    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-nzsqn"
Jul 28 01:34:18.952691 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:34:18.952656    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:19.028235 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:34:19.028205    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:19.099279 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:34:19.099247    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:19.199787 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:34:19.199737    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:19.557924 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:34:19.557893    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:19.616914 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:34:19.616884    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:56.387090 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:34:56.387052    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:54.341089 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:35:54.341055    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:54.441886 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:35:54.441852    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:06.592296 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.592240    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:06.718551 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.718515    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"]
Jul 28 01:36:06.718773 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.718761    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88705942-2637-4b34-a67a-3cc1b9c5442a" containerName="manager"
Jul 28 01:36:06.718828 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.718774    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="88705942-2637-4b34-a67a-3cc1b9c5442a" containerName="manager"
Jul 28 01:36:06.718828 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.718814    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="88705942-2637-4b34-a67a-3cc1b9c5442a" containerName="manager"
Jul 28 01:36:06.721715 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.721699    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:06.723904 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.723884    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a4889-predictor-serving-cert\""
Jul 28 01:36:06.724125 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.724108    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 01:36:06.724257 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.724237    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 01:36:06.724324 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.724279    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 01:36:06.724427 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.724411    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 01:36:06.724609 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.724591    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a4889-kube-rbac-proxy-sar-config\""
Jul 28 01:36:06.725048 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.725032    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cl94s\""
Jul 28 01:36:06.731648 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.731628    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"]
Jul 28 01:36:06.763587 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.763549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23545a18-dca2-4741-bae5-e7ecaf395f63-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:06.763587 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.763589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23545a18-dca2-4741-bae5-e7ecaf395f63-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:06.763783 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.763623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23545a18-dca2-4741-bae5-e7ecaf395f63-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:06.763783 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.763651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23545a18-dca2-4741-bae5-e7ecaf395f63-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:06.763783 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.763677    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq9t5\" (UniqueName: \"kubernetes.io/projected/23545a18-dca2-4741-bae5-e7ecaf395f63-kube-api-access-vq9t5\") pod \"isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:06.857710 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.857621    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:06.864622 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.864594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23545a18-dca2-4741-bae5-e7ecaf395f63-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:06.864762 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.864640    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23545a18-dca2-4741-bae5-e7ecaf395f63-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:06.864762 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.864684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23545a18-dca2-4741-bae5-e7ecaf395f63-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:06.864762 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.864729    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23545a18-dca2-4741-bae5-e7ecaf395f63-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:06.864928 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.864773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq9t5\" (UniqueName: \"kubernetes.io/projected/23545a18-dca2-4741-bae5-e7ecaf395f63-kube-api-access-vq9t5\") pod \"isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:06.865063 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.865042    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23545a18-dca2-4741-bae5-e7ecaf395f63-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:06.865401 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.865367    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23545a18-dca2-4741-bae5-e7ecaf395f63-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:06.865528 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.865414    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23545a18-dca2-4741-bae5-e7ecaf395f63-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:06.867362 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.867344    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23545a18-dca2-4741-bae5-e7ecaf395f63-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:06.873375 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:06.873352    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq9t5\" (UniqueName: \"kubernetes.io/projected/23545a18-dca2-4741-bae5-e7ecaf395f63-kube-api-access-vq9t5\") pod \"isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:07.032503 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:07.032464    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:07.163761 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:07.163732    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"]
Jul 28 01:36:07.166543 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:36:07.166515    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23545a18_dca2_4741_bae5_e7ecaf395f63.slice/crio-aae75781fc56fec012acdd93be6a521013c92d862849d7f88b4239cd55bd98ac WatchSource:0}: Error finding container aae75781fc56fec012acdd93be6a521013c92d862849d7f88b4239cd55bd98ac: Status 404 returned error can't find the container with id aae75781fc56fec012acdd93be6a521013c92d862849d7f88b4239cd55bd98ac
Jul 28 01:36:07.454240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:07.454156    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" event={"ID":"23545a18-dca2-4741-bae5-e7ecaf395f63","Type":"ContainerStarted","Data":"aae75781fc56fec012acdd93be6a521013c92d862849d7f88b4239cd55bd98ac"}
Jul 28 01:36:10.465527 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:10.465434    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" event={"ID":"23545a18-dca2-4741-bae5-e7ecaf395f63","Type":"ContainerStarted","Data":"7b7ba9e339fa645280956c75a0db7cabe886998317ef037b4fcaf272d9e9b69a"}
Jul 28 01:36:14.480511 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:14.480476    2576 generic.go:356] "Generic (PLEG): container finished" podID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerID="7b7ba9e339fa645280956c75a0db7cabe886998317ef037b4fcaf272d9e9b69a" exitCode=0
Jul 28 01:36:14.480913 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:14.480533    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" event={"ID":"23545a18-dca2-4741-bae5-e7ecaf395f63","Type":"ContainerDied","Data":"7b7ba9e339fa645280956c75a0db7cabe886998317ef037b4fcaf272d9e9b69a"}
Jul 28 01:36:28.542555 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:28.542517    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" event={"ID":"23545a18-dca2-4741-bae5-e7ecaf395f63","Type":"ContainerStarted","Data":"32604feae65926c5587f1988a85723854e40ddce6c8a05d6b677e5520649ae35"}
Jul 28 01:36:30.553892 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:30.553852    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" event={"ID":"23545a18-dca2-4741-bae5-e7ecaf395f63","Type":"ContainerStarted","Data":"459e29983d4b354ce6cc13f8ce892f2e5f021413e8b8d64e129474588f4ea254"}
Jul 28 01:36:32.564215 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:32.564177    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" event={"ID":"23545a18-dca2-4741-bae5-e7ecaf395f63","Type":"ContainerStarted","Data":"b1a2ccd634daf43f9395751f8e1eb22499547bdae8467dd23f75cbe85607d51a"}
Jul 28 01:36:32.564688 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:32.564431    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:32.564688 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:32.564481    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:32.564688 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:32.564537    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:32.565883 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:32.565849    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Jul 28 01:36:32.566527 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:32.566505    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:36:32.584608 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:32.584561    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podStartSLOduration=2.302150515 podStartE2EDuration="26.584546997s" totalImagesPullingTime="20.484453672s" totalInitContainerRuntime="3.79794281s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:36:06 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:36:32.583527095 +0000 UTC m=+622.754566132" watchObservedRunningTime="2026-07-28 01:36:32.584546997 +0000 UTC m=+622.755586033"
Jul 28 01:36:33.568353 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:33.568305    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Jul 28 01:36:33.568821 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:33.568671    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:36:38.572348 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:38.572319    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:36:38.572897 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:38.572861    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Jul 28 01:36:38.573151 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:38.573127    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:36:48.573509 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:48.573457    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Jul 28 01:36:48.573928 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:48.573894    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:36:58.573226 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:58.573176    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Jul 28 01:36:58.573779 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:36:58.573568    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:08.573094 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:08.573042    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Jul 28 01:37:08.573612 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:08.573509    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:18.573007 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:18.572970    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Jul 28 01:37:18.573498 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:18.573318    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:28.572963 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:28.572921    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Jul 28 01:37:28.573365 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:28.573175    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:34.411415 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:34.411359    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:38.573038 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:38.572993    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Jul 28 01:37:38.573481 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:38.573451    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:43.426801 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:43.426772    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:37:43.427241 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:43.427060    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:37:51.779212 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.779176    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:51.837307 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.837267    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"]
Jul 28 01:37:51.837997 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.837932    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container" containerID="cri-o://32604feae65926c5587f1988a85723854e40ddce6c8a05d6b677e5520649ae35" gracePeriod=30
Jul 28 01:37:51.838311 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.838001    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent" containerID="cri-o://b1a2ccd634daf43f9395751f8e1eb22499547bdae8467dd23f75cbe85607d51a" gracePeriod=30
Jul 28 01:37:51.838446 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.838019    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kube-rbac-proxy" containerID="cri-o://459e29983d4b354ce6cc13f8ce892f2e5f021413e8b8d64e129474588f4ea254" gracePeriod=30
Jul 28 01:37:51.839656 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.839633    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"]
Jul 28 01:37:51.842709 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.842687    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:51.845060 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.845039    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-b5420-kube-rbac-proxy-sar-config\""
Jul 28 01:37:51.848708 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.848690    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-b5420-predictor-serving-cert\""
Jul 28 01:37:51.856914 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.856891    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"]
Jul 28 01:37:51.878805 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.878776    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:51.922188 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.922152    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"]
Jul 28 01:37:51.924531 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.924511    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:51.926903 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.926880    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-b5420-predictor-serving-cert\""
Jul 28 01:37:51.926903 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.926898    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-b5420-kube-rbac-proxy-sar-config\""
Jul 28 01:37:51.930171 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.930150    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ceffe1e-148f-4068-b1ab-71795c00c059-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:51.930261 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.930184    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ceffe1e-148f-4068-b1ab-71795c00c059-cabundle-cert\") pod \"isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:51.930261 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.930235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ceffe1e-148f-4068-b1ab-71795c00c059-proxy-tls\") pod \"isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:51.930261 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.930256    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkxk9\" (UniqueName: \"kubernetes.io/projected/8ceffe1e-148f-4068-b1ab-71795c00c059-kube-api-access-bkxk9\") pod \"isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:51.930410 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.930283    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ceffe1e-148f-4068-b1ab-71795c00c059-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:51.935199 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.935166    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"]
Jul 28 01:37:51.959751 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:51.959726    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:52.030855 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.030769    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ac07cff-0fe8-442c-af45-36191e5359d3-proxy-tls\") pod \"isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.030855 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.030815    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ceffe1e-148f-4068-b1ab-71795c00c059-proxy-tls\") pod \"isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:52.030855 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.030844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkxk9\" (UniqueName: \"kubernetes.io/projected/8ceffe1e-148f-4068-b1ab-71795c00c059-kube-api-access-bkxk9\") pod \"isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:52.031138 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.030908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ac07cff-0fe8-442c-af45-36191e5359d3-cabundle-cert\") pod \"isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.031138 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.030962    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ac07cff-0fe8-442c-af45-36191e5359d3-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.031138 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.030995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ceffe1e-148f-4068-b1ab-71795c00c059-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:52.031138 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.031036    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mglsx\" (UniqueName: \"kubernetes.io/projected/6ac07cff-0fe8-442c-af45-36191e5359d3-kube-api-access-mglsx\") pod \"isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.031138 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.031089    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ceffe1e-148f-4068-b1ab-71795c00c059-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:52.031138 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.031114    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ceffe1e-148f-4068-b1ab-71795c00c059-cabundle-cert\") pod \"isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:52.031476 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.031148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ac07cff-0fe8-442c-af45-36191e5359d3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.031476 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.031406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ceffe1e-148f-4068-b1ab-71795c00c059-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:52.031716 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.031695    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ceffe1e-148f-4068-b1ab-71795c00c059-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:52.031779 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.031765    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ceffe1e-148f-4068-b1ab-71795c00c059-cabundle-cert\") pod \"isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:52.033657 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.033641    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ceffe1e-148f-4068-b1ab-71795c00c059-proxy-tls\") pod \"isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:52.039452 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.039430    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkxk9\" (UniqueName: \"kubernetes.io/projected/8ceffe1e-148f-4068-b1ab-71795c00c059-kube-api-access-bkxk9\") pod \"isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:52.058659 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.058619    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:52.132295 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.132257    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ac07cff-0fe8-442c-af45-36191e5359d3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.132295 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.132300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ac07cff-0fe8-442c-af45-36191e5359d3-proxy-tls\") pod \"isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.132537 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.132323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ac07cff-0fe8-442c-af45-36191e5359d3-cabundle-cert\") pod \"isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.132537 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.132415    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ac07cff-0fe8-442c-af45-36191e5359d3-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.132537 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.132461    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mglsx\" (UniqueName: \"kubernetes.io/projected/6ac07cff-0fe8-442c-af45-36191e5359d3-kube-api-access-mglsx\") pod \"isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.132868 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.132836    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ac07cff-0fe8-442c-af45-36191e5359d3-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.133004 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.132986    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ac07cff-0fe8-442c-af45-36191e5359d3-cabundle-cert\") pod \"isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.133141 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.133122    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ac07cff-0fe8-442c-af45-36191e5359d3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.134912 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.134895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ac07cff-0fe8-442c-af45-36191e5359d3-proxy-tls\") pod \"isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.141722 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.141699    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mglsx\" (UniqueName: \"kubernetes.io/projected/6ac07cff-0fe8-442c-af45-36191e5359d3-kube-api-access-mglsx\") pod \"isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.158476 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.158459    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:52.234270 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.234236    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:37:52.291114 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.291079    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"]
Jul 28 01:37:52.371474 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.371449    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"]
Jul 28 01:37:52.374590 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:37:52.374561    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac07cff_0fe8_442c_af45_36191e5359d3.slice/crio-f53d5307fcc1bb6ecd18a8ae1ef54bc891f9908b9b16ade595247e67aa0bc3d2 WatchSource:0}: Error finding container f53d5307fcc1bb6ecd18a8ae1ef54bc891f9908b9b16ade595247e67aa0bc3d2: Status 404 returned error can't find the container with id f53d5307fcc1bb6ecd18a8ae1ef54bc891f9908b9b16ade595247e67aa0bc3d2
Jul 28 01:37:52.836330 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.836287    2576 generic.go:356] "Generic (PLEG): container finished" podID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerID="459e29983d4b354ce6cc13f8ce892f2e5f021413e8b8d64e129474588f4ea254" exitCode=2
Jul 28 01:37:52.836778 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.836358    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" event={"ID":"23545a18-dca2-4741-bae5-e7ecaf395f63","Type":"ContainerDied","Data":"459e29983d4b354ce6cc13f8ce892f2e5f021413e8b8d64e129474588f4ea254"}
Jul 28 01:37:52.838338 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.838311    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" event={"ID":"6ac07cff-0fe8-442c-af45-36191e5359d3","Type":"ContainerStarted","Data":"d653f69b73aa118462725b9702ac759295008098b41c9c80266c54158286702b"}
Jul 28 01:37:52.838475 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.838369    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" event={"ID":"6ac07cff-0fe8-442c-af45-36191e5359d3","Type":"ContainerStarted","Data":"f53d5307fcc1bb6ecd18a8ae1ef54bc891f9908b9b16ade595247e67aa0bc3d2"}
Jul 28 01:37:52.839961 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.839935    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" event={"ID":"8ceffe1e-148f-4068-b1ab-71795c00c059","Type":"ContainerStarted","Data":"9b12034c64d259d13b3e3bdc4c05b75e6ba65e903484a26d0b0fd35fd4e483ba"}
Jul 28 01:37:52.840035 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:52.840002    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" event={"ID":"8ceffe1e-148f-4068-b1ab-71795c00c059","Type":"ContainerStarted","Data":"e953b1fc404e7e5400113d5289a3d3a425f07e5c35cd543e6a2bd2726b087705"}
Jul 28 01:37:53.426430 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:53.426361    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Jul 28 01:37:53.426725 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:53.426698    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:53.569013 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:53.568972    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.15:8643/healthz\": dial tcp 10.134.0.15:8643: connect: connection refused"
Jul 28 01:37:56.856418 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:56.856367    2576 generic.go:356] "Generic (PLEG): container finished" podID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerID="32604feae65926c5587f1988a85723854e40ddce6c8a05d6b677e5520649ae35" exitCode=0
Jul 28 01:37:56.856837 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:56.856421    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" event={"ID":"23545a18-dca2-4741-bae5-e7ecaf395f63","Type":"ContainerDied","Data":"32604feae65926c5587f1988a85723854e40ddce6c8a05d6b677e5520649ae35"}
Jul 28 01:37:56.857853 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:56.857829    2576 generic.go:356] "Generic (PLEG): container finished" podID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerID="d653f69b73aa118462725b9702ac759295008098b41c9c80266c54158286702b" exitCode=0
Jul 28 01:37:56.857967 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:56.857890    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" event={"ID":"6ac07cff-0fe8-442c-af45-36191e5359d3","Type":"ContainerDied","Data":"d653f69b73aa118462725b9702ac759295008098b41c9c80266c54158286702b"}
Jul 28 01:37:56.859330 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:56.859310    2576 generic.go:356] "Generic (PLEG): container finished" podID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerID="9b12034c64d259d13b3e3bdc4c05b75e6ba65e903484a26d0b0fd35fd4e483ba" exitCode=0
Jul 28 01:37:56.859434 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:56.859337    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" event={"ID":"8ceffe1e-148f-4068-b1ab-71795c00c059","Type":"ContainerDied","Data":"9b12034c64d259d13b3e3bdc4c05b75e6ba65e903484a26d0b0fd35fd4e483ba"}
Jul 28 01:37:57.866930 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:57.866832    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" event={"ID":"8ceffe1e-148f-4068-b1ab-71795c00c059","Type":"ContainerStarted","Data":"2bda6091df7868982e5f46a59ee84fc6052a89e0205e4ba04f0fedbedb97d358"}
Jul 28 01:37:57.866930 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:57.866915    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" event={"ID":"8ceffe1e-148f-4068-b1ab-71795c00c059","Type":"ContainerStarted","Data":"29a157e893949f88762d41d6e09b4907b4f9281779aab0709bc606402fb90e99"}
Jul 28 01:37:57.867634 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:57.867557    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:57.867634 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:57.867621    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:37:57.868826 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:57.868791    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 28 01:37:57.895405 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:57.889322    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" podStartSLOduration=3.244117898 podStartE2EDuration="6.889300038s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.64518214s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:37:57.886738104 +0000 UTC m=+708.057777134" watchObservedRunningTime="2026-07-28 01:37:57.889300038 +0000 UTC m=+708.060339077"
Jul 28 01:37:58.569245 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:58.569203    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.15:8643/healthz\": dial tcp 10.134.0.15:8643: connect: connection refused"
Jul 28 01:37:58.872745 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:37:58.872582    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 28 01:38:03.427462 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:03.427192    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Jul 28 01:38:03.427876 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:03.427517    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:03.568777 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:03.568735    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.15:8643/healthz\": dial tcp 10.134.0.15:8643: connect: connection refused"
Jul 28 01:38:03.568982 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:03.568901    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:38:03.877499 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:03.877467    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:38:03.878233 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:03.878191    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 28 01:38:08.569153 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:08.569106    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.15:8643/healthz\": dial tcp 10.134.0.15:8643: connect: connection refused"
Jul 28 01:38:13.427055 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:13.427010    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Jul 28 01:38:13.427535 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:13.427163    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:38:13.427535 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:13.427413    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:13.427535 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:13.427519    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:38:13.568486 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:13.568441    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.15:8643/healthz\": dial tcp 10.134.0.15:8643: connect: connection refused"
Jul 28 01:38:13.878593 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:13.878555    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 28 01:38:15.947331 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:15.947298    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" event={"ID":"6ac07cff-0fe8-442c-af45-36191e5359d3","Type":"ContainerStarted","Data":"9055ab8649db85a98bd43bf3b8c2ddc2de4de8600852ff91b12af2e33d342dac"}
Jul 28 01:38:15.947759 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:15.947360    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" event={"ID":"6ac07cff-0fe8-442c-af45-36191e5359d3","Type":"ContainerStarted","Data":"f96c25a1ca2a80f4e8ddbf487e11cac10887a226dacc2abf0ca4cd6adc7b0eeb"}
Jul 28 01:38:15.947759 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:15.947657    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:38:15.947759 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:15.947705    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:38:15.948866 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:15.948832    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 28 01:38:15.966170 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:15.966055    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" podStartSLOduration=3.143498613 podStartE2EDuration="24.96603824s" totalImagesPullingTime="18.018737286s" totalInitContainerRuntime="3.803802341s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:38:15.965596779 +0000 UTC m=+726.136635826" watchObservedRunningTime="2026-07-28 01:38:15.96603824 +0000 UTC m=+726.137077277"
Jul 28 01:38:16.951260 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:16.951212    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 28 01:38:18.568780 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:18.568734    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.15:8643/healthz\": dial tcp 10.134.0.15:8643: connect: connection refused"
Jul 28 01:38:21.955685 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:21.955658    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:38:21.956331 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:21.956296    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 28 01:38:22.521581 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.521553    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:38:22.597842 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.597814    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23545a18-dca2-4741-bae5-e7ecaf395f63-kserve-provision-location\") pod \"23545a18-dca2-4741-bae5-e7ecaf395f63\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") "
Jul 28 01:38:22.597842 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.597849    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23545a18-dca2-4741-bae5-e7ecaf395f63-proxy-tls\") pod \"23545a18-dca2-4741-bae5-e7ecaf395f63\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") "
Jul 28 01:38:22.598097 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.597872    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23545a18-dca2-4741-bae5-e7ecaf395f63-cabundle-cert\") pod \"23545a18-dca2-4741-bae5-e7ecaf395f63\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") "
Jul 28 01:38:22.598097 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.597911    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23545a18-dca2-4741-bae5-e7ecaf395f63-kube-rbac-proxy-sar-config\") pod \"23545a18-dca2-4741-bae5-e7ecaf395f63\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") "
Jul 28 01:38:22.598097 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.597963    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq9t5\" (UniqueName: \"kubernetes.io/projected/23545a18-dca2-4741-bae5-e7ecaf395f63-kube-api-access-vq9t5\") pod \"23545a18-dca2-4741-bae5-e7ecaf395f63\" (UID: \"23545a18-dca2-4741-bae5-e7ecaf395f63\") "
Jul 28 01:38:22.598262 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.598213    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23545a18-dca2-4741-bae5-e7ecaf395f63-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "23545a18-dca2-4741-bae5-e7ecaf395f63" (UID: "23545a18-dca2-4741-bae5-e7ecaf395f63"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:38:22.598262 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.598245    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23545a18-dca2-4741-bae5-e7ecaf395f63-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "23545a18-dca2-4741-bae5-e7ecaf395f63" (UID: "23545a18-dca2-4741-bae5-e7ecaf395f63"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:38:22.598262 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.598251    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23545a18-dca2-4741-bae5-e7ecaf395f63-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "23545a18-dca2-4741-bae5-e7ecaf395f63" (UID: "23545a18-dca2-4741-bae5-e7ecaf395f63"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:38:22.600271 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.600251    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23545a18-dca2-4741-bae5-e7ecaf395f63-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "23545a18-dca2-4741-bae5-e7ecaf395f63" (UID: "23545a18-dca2-4741-bae5-e7ecaf395f63"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:38:22.600330 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.600286    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23545a18-dca2-4741-bae5-e7ecaf395f63-kube-api-access-vq9t5" (OuterVolumeSpecName: "kube-api-access-vq9t5") pod "23545a18-dca2-4741-bae5-e7ecaf395f63" (UID: "23545a18-dca2-4741-bae5-e7ecaf395f63"). InnerVolumeSpecName "kube-api-access-vq9t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:38:22.698777 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.698749    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23545a18-dca2-4741-bae5-e7ecaf395f63-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:38:22.698777 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.698770    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23545a18-dca2-4741-bae5-e7ecaf395f63-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:38:22.698777 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.698779    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23545a18-dca2-4741-bae5-e7ecaf395f63-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:38:22.698777 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.698787    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23545a18-dca2-4741-bae5-e7ecaf395f63-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:38:22.699024 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.698795    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq9t5\" (UniqueName: \"kubernetes.io/projected/23545a18-dca2-4741-bae5-e7ecaf395f63-kube-api-access-vq9t5\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:38:22.974350 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.974259    2576 generic.go:356] "Generic (PLEG): container finished" podID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerID="b1a2ccd634daf43f9395751f8e1eb22499547bdae8467dd23f75cbe85607d51a" exitCode=0
Jul 28 01:38:22.974350 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.974298    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" event={"ID":"23545a18-dca2-4741-bae5-e7ecaf395f63","Type":"ContainerDied","Data":"b1a2ccd634daf43f9395751f8e1eb22499547bdae8467dd23f75cbe85607d51a"}
Jul 28 01:38:22.974350 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.974330    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"
Jul 28 01:38:22.974350 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.974352    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw" event={"ID":"23545a18-dca2-4741-bae5-e7ecaf395f63","Type":"ContainerDied","Data":"aae75781fc56fec012acdd93be6a521013c92d862849d7f88b4239cd55bd98ac"}
Jul 28 01:38:22.974981 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.974401    2576 scope.go:117] "RemoveContainer" containerID="b1a2ccd634daf43f9395751f8e1eb22499547bdae8467dd23f75cbe85607d51a"
Jul 28 01:38:22.984321 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.984298    2576 scope.go:117] "RemoveContainer" containerID="459e29983d4b354ce6cc13f8ce892f2e5f021413e8b8d64e129474588f4ea254"
Jul 28 01:38:22.991804 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.991785    2576 scope.go:117] "RemoveContainer" containerID="32604feae65926c5587f1988a85723854e40ddce6c8a05d6b677e5520649ae35"
Jul 28 01:38:22.998513 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:22.998498    2576 scope.go:117] "RemoveContainer" containerID="7b7ba9e339fa645280956c75a0db7cabe886998317ef037b4fcaf272d9e9b69a"
Jul 28 01:38:23.001746 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:23.001727    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"]
Jul 28 01:38:23.005630 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:23.005608    2576 scope.go:117] "RemoveContainer" containerID="b1a2ccd634daf43f9395751f8e1eb22499547bdae8467dd23f75cbe85607d51a"
Jul 28 01:38:23.005898 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:38:23.005881    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a2ccd634daf43f9395751f8e1eb22499547bdae8467dd23f75cbe85607d51a\": container with ID starting with b1a2ccd634daf43f9395751f8e1eb22499547bdae8467dd23f75cbe85607d51a not found: ID does not exist" containerID="b1a2ccd634daf43f9395751f8e1eb22499547bdae8467dd23f75cbe85607d51a"
Jul 28 01:38:23.005942 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:23.005908    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a2ccd634daf43f9395751f8e1eb22499547bdae8467dd23f75cbe85607d51a"} err="failed to get container status \"b1a2ccd634daf43f9395751f8e1eb22499547bdae8467dd23f75cbe85607d51a\": rpc error: code = NotFound desc = could not find container \"b1a2ccd634daf43f9395751f8e1eb22499547bdae8467dd23f75cbe85607d51a\": container with ID starting with b1a2ccd634daf43f9395751f8e1eb22499547bdae8467dd23f75cbe85607d51a not found: ID does not exist"
Jul 28 01:38:23.005942 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:23.005926    2576 scope.go:117] "RemoveContainer" containerID="459e29983d4b354ce6cc13f8ce892f2e5f021413e8b8d64e129474588f4ea254"
Jul 28 01:38:23.006220 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:38:23.006203    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459e29983d4b354ce6cc13f8ce892f2e5f021413e8b8d64e129474588f4ea254\": container with ID starting with 459e29983d4b354ce6cc13f8ce892f2e5f021413e8b8d64e129474588f4ea254 not found: ID does not exist" containerID="459e29983d4b354ce6cc13f8ce892f2e5f021413e8b8d64e129474588f4ea254"
Jul 28 01:38:23.006287 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:23.006237    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459e29983d4b354ce6cc13f8ce892f2e5f021413e8b8d64e129474588f4ea254"} err="failed to get container status \"459e29983d4b354ce6cc13f8ce892f2e5f021413e8b8d64e129474588f4ea254\": rpc error: code = NotFound desc = could not find container \"459e29983d4b354ce6cc13f8ce892f2e5f021413e8b8d64e129474588f4ea254\": container with ID starting with 459e29983d4b354ce6cc13f8ce892f2e5f021413e8b8d64e129474588f4ea254 not found: ID does not exist"
Jul 28 01:38:23.006287 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:23.006254    2576 scope.go:117] "RemoveContainer" containerID="32604feae65926c5587f1988a85723854e40ddce6c8a05d6b677e5520649ae35"
Jul 28 01:38:23.006798 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:38:23.006693    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32604feae65926c5587f1988a85723854e40ddce6c8a05d6b677e5520649ae35\": container with ID starting with 32604feae65926c5587f1988a85723854e40ddce6c8a05d6b677e5520649ae35 not found: ID does not exist" containerID="32604feae65926c5587f1988a85723854e40ddce6c8a05d6b677e5520649ae35"
Jul 28 01:38:23.006798 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:23.006751    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32604feae65926c5587f1988a85723854e40ddce6c8a05d6b677e5520649ae35"} err="failed to get container status \"32604feae65926c5587f1988a85723854e40ddce6c8a05d6b677e5520649ae35\": rpc error: code = NotFound desc = could not find container \"32604feae65926c5587f1988a85723854e40ddce6c8a05d6b677e5520649ae35\": container with ID starting with 32604feae65926c5587f1988a85723854e40ddce6c8a05d6b677e5520649ae35 not found: ID does not exist"
Jul 28 01:38:23.006798 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:23.006773    2576 scope.go:117] "RemoveContainer" containerID="7b7ba9e339fa645280956c75a0db7cabe886998317ef037b4fcaf272d9e9b69a"
Jul 28 01:38:23.007114 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:38:23.007090    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b7ba9e339fa645280956c75a0db7cabe886998317ef037b4fcaf272d9e9b69a\": container with ID starting with 7b7ba9e339fa645280956c75a0db7cabe886998317ef037b4fcaf272d9e9b69a not found: ID does not exist" containerID="7b7ba9e339fa645280956c75a0db7cabe886998317ef037b4fcaf272d9e9b69a"
Jul 28 01:38:23.007181 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:23.007121    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b7ba9e339fa645280956c75a0db7cabe886998317ef037b4fcaf272d9e9b69a"} err="failed to get container status \"7b7ba9e339fa645280956c75a0db7cabe886998317ef037b4fcaf272d9e9b69a\": rpc error: code = NotFound desc = could not find container \"7b7ba9e339fa645280956c75a0db7cabe886998317ef037b4fcaf272d9e9b69a\": container with ID starting with 7b7ba9e339fa645280956c75a0db7cabe886998317ef037b4fcaf272d9e9b69a not found: ID does not exist"
Jul 28 01:38:23.008328 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:23.008310    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a4889-predictor-6c6f67fb9b-8dmxw"]
Jul 28 01:38:23.878777 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:23.878742    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 28 01:38:24.431474 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:24.431442    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" path="/var/lib/kubelet/pods/23545a18-dca2-4741-bae5-e7ecaf395f63/volumes"
Jul 28 01:38:31.956242 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:31.956201    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 28 01:38:33.878212 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:33.878164    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 28 01:38:41.956671 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:41.956618    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 28 01:38:43.878749 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:43.878701    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 28 01:38:51.956416 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:51.956359    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 28 01:38:53.878685 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:38:53.878649    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 28 01:39:01.956838 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:01.956798    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 28 01:39:03.878698 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:03.878654    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 28 01:39:08.426590 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:08.426548    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 28 01:39:11.956821 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:11.956780    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 28 01:39:18.431182 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:18.431155    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:39:21.956561 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:21.956535    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:39:31.915595 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:31.915511    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:32.254790 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:32.254755    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:41.933570 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:41.933538    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:42.098045 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.098012    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"]
Jul 28 01:39:42.098312 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.098267    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kserve-container" containerID="cri-o://29a157e893949f88762d41d6e09b4907b4f9281779aab0709bc606402fb90e99" gracePeriod=30
Jul 28 01:39:42.098407 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.098350    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kube-rbac-proxy" containerID="cri-o://2bda6091df7868982e5f46a59ee84fc6052a89e0205e4ba04f0fedbedb97d358" gracePeriod=30
Jul 28 01:39:42.102463 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.102374    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:42.168116 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.168088    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"]
Jul 28 01:39:42.168364 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.168342    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kube-rbac-proxy"
Jul 28 01:39:42.168364 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.168359    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kube-rbac-proxy"
Jul 28 01:39:42.168581 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.168374    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent"
Jul 28 01:39:42.168581 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.168403    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent"
Jul 28 01:39:42.168581 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.168428    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container"
Jul 28 01:39:42.168581 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.168437    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container"
Jul 28 01:39:42.168581 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.168453    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="storage-initializer"
Jul 28 01:39:42.168581 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.168462    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="storage-initializer"
Jul 28 01:39:42.168581 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.168519    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="agent"
Jul 28 01:39:42.168581 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.168530    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kube-rbac-proxy"
Jul 28 01:39:42.168581 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.168541    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="23545a18-dca2-4741-bae5-e7ecaf395f63" containerName="kserve-container"
Jul 28 01:39:42.170965 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.170945    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.173412 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.173373    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-cd9f7-kube-rbac-proxy-sar-config\""
Jul 28 01:39:42.173530 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.173376    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-serving-cert\""
Jul 28 01:39:42.183327 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.183306    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"]
Jul 28 01:39:42.189060 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.189007    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:42.249034 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.249005    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"]
Jul 28 01:39:42.252196 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.251458    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.254746 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.254723    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-cd9f7-kube-rbac-proxy-sar-config\""
Jul 28 01:39:42.254896 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.254788    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-serving-cert\""
Jul 28 01:39:42.258625 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.258602    2576 generic.go:356] "Generic (PLEG): container finished" podID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerID="2bda6091df7868982e5f46a59ee84fc6052a89e0205e4ba04f0fedbedb97d358" exitCode=2
Jul 28 01:39:42.258717 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.258633    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" event={"ID":"8ceffe1e-148f-4068-b1ab-71795c00c059","Type":"ContainerDied","Data":"2bda6091df7868982e5f46a59ee84fc6052a89e0205e4ba04f0fedbedb97d358"}
Jul 28 01:39:42.262074 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.262053    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"]
Jul 28 01:39:42.274796 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.274770    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:42.292982 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.292949    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9btb\" (UniqueName: \"kubernetes.io/projected/96b0f7bb-0b37-49df-8970-0e19faf64778-kube-api-access-z9btb\") pod \"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.293141 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.293029    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96b0f7bb-0b37-49df-8970-0e19faf64778-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.293141 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.293073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96b0f7bb-0b37-49df-8970-0e19faf64778-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.293141 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.293127    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96b0f7bb-0b37-49df-8970-0e19faf64778-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.293321 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.293183    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96b0f7bb-0b37-49df-8970-0e19faf64778-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.310519 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.310491    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"]
Jul 28 01:39:42.310758 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.310731    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kserve-container" containerID="cri-o://9055ab8649db85a98bd43bf3b8c2ddc2de4de8600852ff91b12af2e33d342dac" gracePeriod=30
Jul 28 01:39:42.310834 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.310767    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kube-rbac-proxy" containerID="cri-o://f96c25a1ca2a80f4e8ddbf487e11cac10887a226dacc2abf0ca4cd6adc7b0eeb" gracePeriod=30
Jul 28 01:39:42.348081 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.348055    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:42.394244 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.394207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1915c51c-c201-4148-97fb-f5b2b703171b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.394463 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.394249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96b0f7bb-0b37-49df-8970-0e19faf64778-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.394463 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.394270    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bgrt\" (UniqueName: \"kubernetes.io/projected/1915c51c-c201-4148-97fb-f5b2b703171b-kube-api-access-9bgrt\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.394463 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.394349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96b0f7bb-0b37-49df-8970-0e19faf64778-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.394463 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.394417    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96b0f7bb-0b37-49df-8970-0e19faf64778-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.394463 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.394455    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9btb\" (UniqueName: \"kubernetes.io/projected/96b0f7bb-0b37-49df-8970-0e19faf64778-kube-api-access-z9btb\") pod \"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.394752 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.394516    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1915c51c-c201-4148-97fb-f5b2b703171b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.394752 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.394556    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1915c51c-c201-4148-97fb-f5b2b703171b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.394752 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.394625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96b0f7bb-0b37-49df-8970-0e19faf64778-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.394752 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.394671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1915c51c-c201-4148-97fb-f5b2b703171b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.394967 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.394913    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96b0f7bb-0b37-49df-8970-0e19faf64778-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.395076 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.395035    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96b0f7bb-0b37-49df-8970-0e19faf64778-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.395136 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.395097    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96b0f7bb-0b37-49df-8970-0e19faf64778-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.396927 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.396902    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96b0f7bb-0b37-49df-8970-0e19faf64778-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.402984 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.402963    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9btb\" (UniqueName: \"kubernetes.io/projected/96b0f7bb-0b37-49df-8970-0e19faf64778-kube-api-access-z9btb\") pod \"isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.481054 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.480962    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:42.495295 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.495262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1915c51c-c201-4148-97fb-f5b2b703171b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.495459 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.495308    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1915c51c-c201-4148-97fb-f5b2b703171b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.495459 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.495330    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1915c51c-c201-4148-97fb-f5b2b703171b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.495459 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.495350    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bgrt\" (UniqueName: \"kubernetes.io/projected/1915c51c-c201-4148-97fb-f5b2b703171b-kube-api-access-9bgrt\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.495459 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.495424    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1915c51c-c201-4148-97fb-f5b2b703171b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.495683 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:39:42.495497    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-cd9f7-predictor-serving-cert" not found
Jul 28 01:39:42.495683 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:39:42.495567    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1915c51c-c201-4148-97fb-f5b2b703171b-proxy-tls podName:1915c51c-c201-4148-97fb-f5b2b703171b nodeName:}" failed. No retries permitted until 2026-07-28 01:39:42.995545769 +0000 UTC m=+813.166584805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1915c51c-c201-4148-97fb-f5b2b703171b-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" (UID: "1915c51c-c201-4148-97fb-f5b2b703171b") : secret "isvc-xgboost-graph-raw-hpa-cd9f7-predictor-serving-cert" not found
Jul 28 01:39:42.495778 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.495693    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1915c51c-c201-4148-97fb-f5b2b703171b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.495971 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.495954    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1915c51c-c201-4148-97fb-f5b2b703171b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.496040 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.496023    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1915c51c-c201-4148-97fb-f5b2b703171b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.504352 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.504330    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bgrt\" (UniqueName: \"kubernetes.io/projected/1915c51c-c201-4148-97fb-f5b2b703171b-kube-api-access-9bgrt\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.609224 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.609199    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"]
Jul 28 01:39:42.611577 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:39:42.611550    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b0f7bb_0b37_49df_8970_0e19faf64778.slice/crio-3843e083363040b7599fbc64d66b80fa9a03a7e2d361485091b06ccfa183028b WatchSource:0}: Error finding container 3843e083363040b7599fbc64d66b80fa9a03a7e2d361485091b06ccfa183028b: Status 404 returned error can't find the container with id 3843e083363040b7599fbc64d66b80fa9a03a7e2d361485091b06ccfa183028b
Jul 28 01:39:42.999001 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:42.998966    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1915c51c-c201-4148-97fb-f5b2b703171b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:42.999412 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:39:42.999157    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-cd9f7-predictor-serving-cert" not found
Jul 28 01:39:42.999412 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:39:42.999248    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1915c51c-c201-4148-97fb-f5b2b703171b-proxy-tls podName:1915c51c-c201-4148-97fb-f5b2b703171b nodeName:}" failed. No retries permitted until 2026-07-28 01:39:43.999224962 +0000 UTC m=+814.170263996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1915c51c-c201-4148-97fb-f5b2b703171b-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" (UID: "1915c51c-c201-4148-97fb-f5b2b703171b") : secret "isvc-xgboost-graph-raw-hpa-cd9f7-predictor-serving-cert" not found
Jul 28 01:39:43.212825 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:43.212791    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:43.264724 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:43.264641    2576 generic.go:356] "Generic (PLEG): container finished" podID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerID="f96c25a1ca2a80f4e8ddbf487e11cac10887a226dacc2abf0ca4cd6adc7b0eeb" exitCode=2
Jul 28 01:39:43.264878 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:43.264720    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" event={"ID":"6ac07cff-0fe8-442c-af45-36191e5359d3","Type":"ContainerDied","Data":"f96c25a1ca2a80f4e8ddbf487e11cac10887a226dacc2abf0ca4cd6adc7b0eeb"}
Jul 28 01:39:43.265946 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:43.265929    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" event={"ID":"96b0f7bb-0b37-49df-8970-0e19faf64778","Type":"ContainerStarted","Data":"2356ffb0209706d16538c6ca969527c6b7c4f3c694a0a66327589c2bd66db90d"}
Jul 28 01:39:43.266003 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:43.265968    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" event={"ID":"96b0f7bb-0b37-49df-8970-0e19faf64778","Type":"ContainerStarted","Data":"3843e083363040b7599fbc64d66b80fa9a03a7e2d361485091b06ccfa183028b"}
Jul 28 01:39:43.873660 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:43.873617    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.16:8643/healthz\": dial tcp 10.134.0.16:8643: connect: connection refused"
Jul 28 01:39:44.007699 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:44.007659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1915c51c-c201-4148-97fb-f5b2b703171b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:44.010234 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:44.010206    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1915c51c-c201-4148-97fb-f5b2b703171b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:44.065140 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:44.065092    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:44.191047 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:44.191022    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"]
Jul 28 01:39:44.193433 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:39:44.193375    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1915c51c_c201_4148_97fb_f5b2b703171b.slice/crio-324cf382749563a926b8ee1c781b946f73b4141f0c16223920e6ad8cee50190c WatchSource:0}: Error finding container 324cf382749563a926b8ee1c781b946f73b4141f0c16223920e6ad8cee50190c: Status 404 returned error can't find the container with id 324cf382749563a926b8ee1c781b946f73b4141f0c16223920e6ad8cee50190c
Jul 28 01:39:44.271223 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:44.271183    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" event={"ID":"1915c51c-c201-4148-97fb-f5b2b703171b","Type":"ContainerStarted","Data":"2151539968d438796f2a6e24a5c55348edfb31fea907a9d3a28da5bfdc143e1f"}
Jul 28 01:39:44.271327 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:44.271274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" event={"ID":"1915c51c-c201-4148-97fb-f5b2b703171b","Type":"ContainerStarted","Data":"324cf382749563a926b8ee1c781b946f73b4141f0c16223920e6ad8cee50190c"}
Jul 28 01:39:45.951884 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:45.951862    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:39:46.022130 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.022090    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mglsx\" (UniqueName: \"kubernetes.io/projected/6ac07cff-0fe8-442c-af45-36191e5359d3-kube-api-access-mglsx\") pod \"6ac07cff-0fe8-442c-af45-36191e5359d3\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") "
Jul 28 01:39:46.022308 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.022147    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ac07cff-0fe8-442c-af45-36191e5359d3-cabundle-cert\") pod \"6ac07cff-0fe8-442c-af45-36191e5359d3\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") "
Jul 28 01:39:46.022308 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.022192    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ac07cff-0fe8-442c-af45-36191e5359d3-kserve-provision-location\") pod \"6ac07cff-0fe8-442c-af45-36191e5359d3\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") "
Jul 28 01:39:46.022308 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.022225    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ac07cff-0fe8-442c-af45-36191e5359d3-kube-rbac-proxy-sar-config\") pod \"6ac07cff-0fe8-442c-af45-36191e5359d3\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") "
Jul 28 01:39:46.022308 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.022262    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ac07cff-0fe8-442c-af45-36191e5359d3-proxy-tls\") pod \"6ac07cff-0fe8-442c-af45-36191e5359d3\" (UID: \"6ac07cff-0fe8-442c-af45-36191e5359d3\") "
Jul 28 01:39:46.022591 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.022569    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac07cff-0fe8-442c-af45-36191e5359d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ac07cff-0fe8-442c-af45-36191e5359d3" (UID: "6ac07cff-0fe8-442c-af45-36191e5359d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:39:46.022638 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.022568    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac07cff-0fe8-442c-af45-36191e5359d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ac07cff-0fe8-442c-af45-36191e5359d3" (UID: "6ac07cff-0fe8-442c-af45-36191e5359d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:39:46.022675 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.022646    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac07cff-0fe8-442c-af45-36191e5359d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ac07cff-0fe8-442c-af45-36191e5359d3" (UID: "6ac07cff-0fe8-442c-af45-36191e5359d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:39:46.024495 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.024470    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac07cff-0fe8-442c-af45-36191e5359d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ac07cff-0fe8-442c-af45-36191e5359d3" (UID: "6ac07cff-0fe8-442c-af45-36191e5359d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:39:46.024597 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.024538    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac07cff-0fe8-442c-af45-36191e5359d3-kube-api-access-mglsx" (OuterVolumeSpecName: "kube-api-access-mglsx") pod "6ac07cff-0fe8-442c-af45-36191e5359d3" (UID: "6ac07cff-0fe8-442c-af45-36191e5359d3"). InnerVolumeSpecName "kube-api-access-mglsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:39:46.122734 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.122705    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mglsx\" (UniqueName: \"kubernetes.io/projected/6ac07cff-0fe8-442c-af45-36191e5359d3-kube-api-access-mglsx\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:39:46.122734 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.122732    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ac07cff-0fe8-442c-af45-36191e5359d3-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:39:46.122932 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.122745    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ac07cff-0fe8-442c-af45-36191e5359d3-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:39:46.122932 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.122757    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ac07cff-0fe8-442c-af45-36191e5359d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:39:46.122932 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.122770    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ac07cff-0fe8-442c-af45-36191e5359d3-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:39:46.281964 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.281878    2576 generic.go:356] "Generic (PLEG): container finished" podID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerID="9055ab8649db85a98bd43bf3b8c2ddc2de4de8600852ff91b12af2e33d342dac" exitCode=0
Jul 28 01:39:46.281964 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.281934    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"
Jul 28 01:39:46.282151 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.281962    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" event={"ID":"6ac07cff-0fe8-442c-af45-36191e5359d3","Type":"ContainerDied","Data":"9055ab8649db85a98bd43bf3b8c2ddc2de4de8600852ff91b12af2e33d342dac"}
Jul 28 01:39:46.282151 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.282075    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw" event={"ID":"6ac07cff-0fe8-442c-af45-36191e5359d3","Type":"ContainerDied","Data":"f53d5307fcc1bb6ecd18a8ae1ef54bc891f9908b9b16ade595247e67aa0bc3d2"}
Jul 28 01:39:46.282151 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.282119    2576 scope.go:117] "RemoveContainer" containerID="f96c25a1ca2a80f4e8ddbf487e11cac10887a226dacc2abf0ca4cd6adc7b0eeb"
Jul 28 01:39:46.291977 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.291937    2576 scope.go:117] "RemoveContainer" containerID="9055ab8649db85a98bd43bf3b8c2ddc2de4de8600852ff91b12af2e33d342dac"
Jul 28 01:39:46.300034 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.300014    2576 scope.go:117] "RemoveContainer" containerID="d653f69b73aa118462725b9702ac759295008098b41c9c80266c54158286702b"
Jul 28 01:39:46.306076 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.306056    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"]
Jul 28 01:39:46.309534 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.309514    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b5420-predictor-65476d6885-h8jkw"]
Jul 28 01:39:46.348077 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.348057    2576 scope.go:117] "RemoveContainer" containerID="f96c25a1ca2a80f4e8ddbf487e11cac10887a226dacc2abf0ca4cd6adc7b0eeb"
Jul 28 01:39:46.348367 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:39:46.348348    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f96c25a1ca2a80f4e8ddbf487e11cac10887a226dacc2abf0ca4cd6adc7b0eeb\": container with ID starting with f96c25a1ca2a80f4e8ddbf487e11cac10887a226dacc2abf0ca4cd6adc7b0eeb not found: ID does not exist" containerID="f96c25a1ca2a80f4e8ddbf487e11cac10887a226dacc2abf0ca4cd6adc7b0eeb"
Jul 28 01:39:46.348455 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.348404    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96c25a1ca2a80f4e8ddbf487e11cac10887a226dacc2abf0ca4cd6adc7b0eeb"} err="failed to get container status \"f96c25a1ca2a80f4e8ddbf487e11cac10887a226dacc2abf0ca4cd6adc7b0eeb\": rpc error: code = NotFound desc = could not find container \"f96c25a1ca2a80f4e8ddbf487e11cac10887a226dacc2abf0ca4cd6adc7b0eeb\": container with ID starting with f96c25a1ca2a80f4e8ddbf487e11cac10887a226dacc2abf0ca4cd6adc7b0eeb not found: ID does not exist"
Jul 28 01:39:46.348455 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.348430    2576 scope.go:117] "RemoveContainer" containerID="9055ab8649db85a98bd43bf3b8c2ddc2de4de8600852ff91b12af2e33d342dac"
Jul 28 01:39:46.348687 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:39:46.348671    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9055ab8649db85a98bd43bf3b8c2ddc2de4de8600852ff91b12af2e33d342dac\": container with ID starting with 9055ab8649db85a98bd43bf3b8c2ddc2de4de8600852ff91b12af2e33d342dac not found: ID does not exist" containerID="9055ab8649db85a98bd43bf3b8c2ddc2de4de8600852ff91b12af2e33d342dac"
Jul 28 01:39:46.348731 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.348694    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9055ab8649db85a98bd43bf3b8c2ddc2de4de8600852ff91b12af2e33d342dac"} err="failed to get container status \"9055ab8649db85a98bd43bf3b8c2ddc2de4de8600852ff91b12af2e33d342dac\": rpc error: code = NotFound desc = could not find container \"9055ab8649db85a98bd43bf3b8c2ddc2de4de8600852ff91b12af2e33d342dac\": container with ID starting with 9055ab8649db85a98bd43bf3b8c2ddc2de4de8600852ff91b12af2e33d342dac not found: ID does not exist"
Jul 28 01:39:46.348731 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.348711    2576 scope.go:117] "RemoveContainer" containerID="d653f69b73aa118462725b9702ac759295008098b41c9c80266c54158286702b"
Jul 28 01:39:46.348945 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:39:46.348923    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d653f69b73aa118462725b9702ac759295008098b41c9c80266c54158286702b\": container with ID starting with d653f69b73aa118462725b9702ac759295008098b41c9c80266c54158286702b not found: ID does not exist" containerID="d653f69b73aa118462725b9702ac759295008098b41c9c80266c54158286702b"
Jul 28 01:39:46.349008 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.348948    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d653f69b73aa118462725b9702ac759295008098b41c9c80266c54158286702b"} err="failed to get container status \"d653f69b73aa118462725b9702ac759295008098b41c9c80266c54158286702b\": rpc error: code = NotFound desc = could not find container \"d653f69b73aa118462725b9702ac759295008098b41c9c80266c54158286702b\": container with ID starting with d653f69b73aa118462725b9702ac759295008098b41c9c80266c54158286702b not found: ID does not exist"
Jul 28 01:39:46.430435 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:46.430408    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" path="/var/lib/kubelet/pods/6ac07cff-0fe8-442c-af45-36191e5359d3/volumes"
Jul 28 01:39:47.148642 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.148620    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:39:47.230154 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.230124    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ceffe1e-148f-4068-b1ab-71795c00c059-proxy-tls\") pod \"8ceffe1e-148f-4068-b1ab-71795c00c059\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") "
Jul 28 01:39:47.230417 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.230170    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ceffe1e-148f-4068-b1ab-71795c00c059-kserve-provision-location\") pod \"8ceffe1e-148f-4068-b1ab-71795c00c059\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") "
Jul 28 01:39:47.230417 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.230195    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ceffe1e-148f-4068-b1ab-71795c00c059-cabundle-cert\") pod \"8ceffe1e-148f-4068-b1ab-71795c00c059\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") "
Jul 28 01:39:47.230417 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.230213    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkxk9\" (UniqueName: \"kubernetes.io/projected/8ceffe1e-148f-4068-b1ab-71795c00c059-kube-api-access-bkxk9\") pod \"8ceffe1e-148f-4068-b1ab-71795c00c059\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") "
Jul 28 01:39:47.230417 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.230236    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ceffe1e-148f-4068-b1ab-71795c00c059-kube-rbac-proxy-sar-config\") pod \"8ceffe1e-148f-4068-b1ab-71795c00c059\" (UID: \"8ceffe1e-148f-4068-b1ab-71795c00c059\") "
Jul 28 01:39:47.230642 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.230602    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ceffe1e-148f-4068-b1ab-71795c00c059-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ceffe1e-148f-4068-b1ab-71795c00c059" (UID: "8ceffe1e-148f-4068-b1ab-71795c00c059"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:39:47.230642 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.230630    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ceffe1e-148f-4068-b1ab-71795c00c059-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ceffe1e-148f-4068-b1ab-71795c00c059" (UID: "8ceffe1e-148f-4068-b1ab-71795c00c059"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:39:47.230740 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.230687    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ceffe1e-148f-4068-b1ab-71795c00c059-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ceffe1e-148f-4068-b1ab-71795c00c059" (UID: "8ceffe1e-148f-4068-b1ab-71795c00c059"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:39:47.232418 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.232395    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ceffe1e-148f-4068-b1ab-71795c00c059-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ceffe1e-148f-4068-b1ab-71795c00c059" (UID: "8ceffe1e-148f-4068-b1ab-71795c00c059"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:39:47.232418 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.232410    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ceffe1e-148f-4068-b1ab-71795c00c059-kube-api-access-bkxk9" (OuterVolumeSpecName: "kube-api-access-bkxk9") pod "8ceffe1e-148f-4068-b1ab-71795c00c059" (UID: "8ceffe1e-148f-4068-b1ab-71795c00c059"). InnerVolumeSpecName "kube-api-access-bkxk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:39:47.287085 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.287049    2576 generic.go:356] "Generic (PLEG): container finished" podID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerID="2356ffb0209706d16538c6ca969527c6b7c4f3c694a0a66327589c2bd66db90d" exitCode=0
Jul 28 01:39:47.287261 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.287125    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" event={"ID":"96b0f7bb-0b37-49df-8970-0e19faf64778","Type":"ContainerDied","Data":"2356ffb0209706d16538c6ca969527c6b7c4f3c694a0a66327589c2bd66db90d"}
Jul 28 01:39:47.289865 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.289842    2576 generic.go:356] "Generic (PLEG): container finished" podID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerID="29a157e893949f88762d41d6e09b4907b4f9281779aab0709bc606402fb90e99" exitCode=0
Jul 28 01:39:47.289940 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.289881    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" event={"ID":"8ceffe1e-148f-4068-b1ab-71795c00c059","Type":"ContainerDied","Data":"29a157e893949f88762d41d6e09b4907b4f9281779aab0709bc606402fb90e99"}
Jul 28 01:39:47.289940 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.289891    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"
Jul 28 01:39:47.289940 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.289937    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82" event={"ID":"8ceffe1e-148f-4068-b1ab-71795c00c059","Type":"ContainerDied","Data":"e953b1fc404e7e5400113d5289a3d3a425f07e5c35cd543e6a2bd2726b087705"}
Jul 28 01:39:47.290038 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.289975    2576 scope.go:117] "RemoveContainer" containerID="2bda6091df7868982e5f46a59ee84fc6052a89e0205e4ba04f0fedbedb97d358"
Jul 28 01:39:47.300229 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.300207    2576 scope.go:117] "RemoveContainer" containerID="29a157e893949f88762d41d6e09b4907b4f9281779aab0709bc606402fb90e99"
Jul 28 01:39:47.308190 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.308168    2576 scope.go:117] "RemoveContainer" containerID="9b12034c64d259d13b3e3bdc4c05b75e6ba65e903484a26d0b0fd35fd4e483ba"
Jul 28 01:39:47.319927 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.319878    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"]
Jul 28 01:39:47.326034 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.325900    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-b5420-predictor-6795699964-7vb82"]
Jul 28 01:39:47.329889 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.326130    2576 scope.go:117] "RemoveContainer" containerID="2bda6091df7868982e5f46a59ee84fc6052a89e0205e4ba04f0fedbedb97d358"
Jul 28 01:39:47.330234 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:39:47.330198    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bda6091df7868982e5f46a59ee84fc6052a89e0205e4ba04f0fedbedb97d358\": container with ID starting with 2bda6091df7868982e5f46a59ee84fc6052a89e0205e4ba04f0fedbedb97d358 not found: ID does not exist" containerID="2bda6091df7868982e5f46a59ee84fc6052a89e0205e4ba04f0fedbedb97d358"
Jul 28 01:39:47.330537 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.330456    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bda6091df7868982e5f46a59ee84fc6052a89e0205e4ba04f0fedbedb97d358"} err="failed to get container status \"2bda6091df7868982e5f46a59ee84fc6052a89e0205e4ba04f0fedbedb97d358\": rpc error: code = NotFound desc = could not find container \"2bda6091df7868982e5f46a59ee84fc6052a89e0205e4ba04f0fedbedb97d358\": container with ID starting with 2bda6091df7868982e5f46a59ee84fc6052a89e0205e4ba04f0fedbedb97d358 not found: ID does not exist"
Jul 28 01:39:47.330537 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.330491    2576 scope.go:117] "RemoveContainer" containerID="29a157e893949f88762d41d6e09b4907b4f9281779aab0709bc606402fb90e99"
Jul 28 01:39:47.331141 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:39:47.331110    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a157e893949f88762d41d6e09b4907b4f9281779aab0709bc606402fb90e99\": container with ID starting with 29a157e893949f88762d41d6e09b4907b4f9281779aab0709bc606402fb90e99 not found: ID does not exist" containerID="29a157e893949f88762d41d6e09b4907b4f9281779aab0709bc606402fb90e99"
Jul 28 01:39:47.331236 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.331145    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a157e893949f88762d41d6e09b4907b4f9281779aab0709bc606402fb90e99"} err="failed to get container status \"29a157e893949f88762d41d6e09b4907b4f9281779aab0709bc606402fb90e99\": rpc error: code = NotFound desc = could not find container \"29a157e893949f88762d41d6e09b4907b4f9281779aab0709bc606402fb90e99\": container with ID starting with 29a157e893949f88762d41d6e09b4907b4f9281779aab0709bc606402fb90e99 not found: ID does not exist"
Jul 28 01:39:47.331236 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.331168    2576 scope.go:117] "RemoveContainer" containerID="9b12034c64d259d13b3e3bdc4c05b75e6ba65e903484a26d0b0fd35fd4e483ba"
Jul 28 01:39:47.331236 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.331194    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkxk9\" (UniqueName: \"kubernetes.io/projected/8ceffe1e-148f-4068-b1ab-71795c00c059-kube-api-access-bkxk9\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:39:47.331236 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.331213    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ceffe1e-148f-4068-b1ab-71795c00c059-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:39:47.331236 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.331229    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ceffe1e-148f-4068-b1ab-71795c00c059-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:39:47.331539 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.331244    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ceffe1e-148f-4068-b1ab-71795c00c059-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:39:47.331539 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.331258    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ceffe1e-148f-4068-b1ab-71795c00c059-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:39:47.331718 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:39:47.331700    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b12034c64d259d13b3e3bdc4c05b75e6ba65e903484a26d0b0fd35fd4e483ba\": container with ID starting with 9b12034c64d259d13b3e3bdc4c05b75e6ba65e903484a26d0b0fd35fd4e483ba not found: ID does not exist" containerID="9b12034c64d259d13b3e3bdc4c05b75e6ba65e903484a26d0b0fd35fd4e483ba"
Jul 28 01:39:47.331772 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:47.331724    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b12034c64d259d13b3e3bdc4c05b75e6ba65e903484a26d0b0fd35fd4e483ba"} err="failed to get container status \"9b12034c64d259d13b3e3bdc4c05b75e6ba65e903484a26d0b0fd35fd4e483ba\": rpc error: code = NotFound desc = could not find container \"9b12034c64d259d13b3e3bdc4c05b75e6ba65e903484a26d0b0fd35fd4e483ba\": container with ID starting with 9b12034c64d259d13b3e3bdc4c05b75e6ba65e903484a26d0b0fd35fd4e483ba not found: ID does not exist"
Jul 28 01:39:48.297113 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:48.297075    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" event={"ID":"96b0f7bb-0b37-49df-8970-0e19faf64778","Type":"ContainerStarted","Data":"e57f350804f7c64d2f4b9c5f4aaca66e717a0e1d95a5512acc9eb818ca53c700"}
Jul 28 01:39:48.297569 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:48.297174    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" event={"ID":"96b0f7bb-0b37-49df-8970-0e19faf64778","Type":"ContainerStarted","Data":"5b365c26a86bbc1c5ca0a364ee3b05aaa89fdf07c034a497879b6281ea076ebd"}
Jul 28 01:39:48.297569 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:48.297365    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:48.298619 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:48.298596    2576 generic.go:356] "Generic (PLEG): container finished" podID="1915c51c-c201-4148-97fb-f5b2b703171b" containerID="2151539968d438796f2a6e24a5c55348edfb31fea907a9d3a28da5bfdc143e1f" exitCode=0
Jul 28 01:39:48.298716 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:48.298632    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" event={"ID":"1915c51c-c201-4148-97fb-f5b2b703171b","Type":"ContainerDied","Data":"2151539968d438796f2a6e24a5c55348edfb31fea907a9d3a28da5bfdc143e1f"}
Jul 28 01:39:48.315752 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:48.315706    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" podStartSLOduration=2.679340052 podStartE2EDuration="6.315686817s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.636346765s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:39:48.315315845 +0000 UTC m=+818.486354882" watchObservedRunningTime="2026-07-28 01:39:48.315686817 +0000 UTC m=+818.486725859"
Jul 28 01:39:48.433282 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:48.433250    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" path="/var/lib/kubelet/pods/8ceffe1e-148f-4068-b1ab-71795c00c059/volumes"
Jul 28 01:39:49.305320 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:49.305286    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" event={"ID":"1915c51c-c201-4148-97fb-f5b2b703171b","Type":"ContainerStarted","Data":"7f7bf86ccffa31eafc64377bd6ea476b4663569c9cbce06c268a96c8e899c766"}
Jul 28 01:39:49.305782 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:49.305343    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" event={"ID":"1915c51c-c201-4148-97fb-f5b2b703171b","Type":"ContainerStarted","Data":"48c41ec4f0abad4a09f369c3eb561dbe9e9d4069ec85b7f0f0b007584c1d6c28"}
Jul 28 01:39:49.305782 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:49.305554    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:49.305782 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:49.305728    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:49.306808 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:49.306781    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 01:39:49.326248 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:49.326207    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" podStartSLOduration=3.7102285 podStartE2EDuration="7.326195473s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.615966973s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:39:49.324489914 +0000 UTC m=+819.495528950" watchObservedRunningTime="2026-07-28 01:39:49.326195473 +0000 UTC m=+819.497234510"
Jul 28 01:39:50.308698 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:50.308652    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 01:39:50.309166 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:50.308808    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:50.309893 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:50.309877    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 01:39:51.312273 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:51.312235    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 01:39:55.313642 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:55.313609    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:39:55.314254 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:55.314224    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 01:39:56.316742 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:56.316709    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:39:56.317185 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:39:56.317159    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 01:40:05.314417 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:40:05.314357    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 01:40:06.317608 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:40:06.317561    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 01:40:15.314699 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:40:15.314658    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 01:40:16.318189 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:40:16.318148    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 01:40:25.314476 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:40:25.314433    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 01:40:26.318076 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:40:26.318034    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 01:40:35.314695 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:40:35.314654    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 01:40:36.317425 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:40:36.317361    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 01:40:45.314931 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:40:45.314888    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 01:40:46.317799 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:40:46.317763    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:40:55.314960 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:40:55.314923    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 01:41:05.315229 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:05.315152    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:41:22.244806 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:22.244770    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:22.653189 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:22.653101    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:31.816790 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:31.816735    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:32.272845 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.272809    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:32.449020 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.448986    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:32.498535 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.498498    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"]
Jul 28 01:41:32.498870 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.498853    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kserve-container"
Jul 28 01:41:32.498959 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.498873    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kserve-container"
Jul 28 01:41:32.498959 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.498894    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kube-rbac-proxy"
Jul 28 01:41:32.498959 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.498902    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kube-rbac-proxy"
Jul 28 01:41:32.498959 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.498914    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kserve-container"
Jul 28 01:41:32.498959 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.498923    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kserve-container"
Jul 28 01:41:32.498959 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.498939    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="storage-initializer"
Jul 28 01:41:32.498959 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.498948    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="storage-initializer"
Jul 28 01:41:32.498959 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.498958    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kube-rbac-proxy"
Jul 28 01:41:32.499333 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.498967    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kube-rbac-proxy"
Jul 28 01:41:32.499333 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.498984    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="storage-initializer"
Jul 28 01:41:32.499333 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.498993    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="storage-initializer"
Jul 28 01:41:32.499333 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.499058    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kube-rbac-proxy"
Jul 28 01:41:32.499333 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.499068    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kserve-container"
Jul 28 01:41:32.499333 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.499076    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ceffe1e-148f-4068-b1ab-71795c00c059" containerName="kserve-container"
Jul 28 01:41:32.499333 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.499085    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac07cff-0fe8-442c-af45-36191e5359d3" containerName="kube-rbac-proxy"
Jul 28 01:41:32.502233 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.502208    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:41:32.506129 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.506101    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-18035-kube-rbac-proxy-sar-config\""
Jul 28 01:41:32.506626 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.506607    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-18035-predictor-serving-cert\""
Jul 28 01:41:32.520047 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.519759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70bedd5b-7206-41fe-b0a0-f19456abe21f-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-18035-predictor-5fdc678c5-z569k\" (UID: \"70bedd5b-7206-41fe-b0a0-f19456abe21f\") " pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:41:32.520047 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.519817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fr8c\" (UniqueName: \"kubernetes.io/projected/70bedd5b-7206-41fe-b0a0-f19456abe21f-kube-api-access-9fr8c\") pod \"message-dumper-raw-18035-predictor-5fdc678c5-z569k\" (UID: \"70bedd5b-7206-41fe-b0a0-f19456abe21f\") " pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:41:32.520047 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.519878    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70bedd5b-7206-41fe-b0a0-f19456abe21f-proxy-tls\") pod \"message-dumper-raw-18035-predictor-5fdc678c5-z569k\" (UID: \"70bedd5b-7206-41fe-b0a0-f19456abe21f\") " pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:41:32.521184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.521162    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"]
Jul 28 01:41:32.558717 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.558626    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"]
Jul 28 01:41:32.558951 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.558922    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kserve-container" containerID="cri-o://5b365c26a86bbc1c5ca0a364ee3b05aaa89fdf07c034a497879b6281ea076ebd" gracePeriod=30
Jul 28 01:41:32.559064 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.558971    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kube-rbac-proxy" containerID="cri-o://e57f350804f7c64d2f4b9c5f4aaca66e717a0e1d95a5512acc9eb818ca53c700" gracePeriod=30
Jul 28 01:41:32.565060 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.565034    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:32.601438 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.601409    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"]
Jul 28 01:41:32.601670 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.601645    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kserve-container" containerID="cri-o://48c41ec4f0abad4a09f369c3eb561dbe9e9d4069ec85b7f0f0b007584c1d6c28" gracePeriod=30
Jul 28 01:41:32.601741 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.601677    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kube-rbac-proxy" containerID="cri-o://7f7bf86ccffa31eafc64377bd6ea476b4663569c9cbce06c268a96c8e899c766" gracePeriod=30
Jul 28 01:41:32.620904 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.620877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fr8c\" (UniqueName: \"kubernetes.io/projected/70bedd5b-7206-41fe-b0a0-f19456abe21f-kube-api-access-9fr8c\") pod \"message-dumper-raw-18035-predictor-5fdc678c5-z569k\" (UID: \"70bedd5b-7206-41fe-b0a0-f19456abe21f\") " pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:41:32.621010 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.620935    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70bedd5b-7206-41fe-b0a0-f19456abe21f-proxy-tls\") pod \"message-dumper-raw-18035-predictor-5fdc678c5-z569k\" (UID: \"70bedd5b-7206-41fe-b0a0-f19456abe21f\") " pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:41:32.621061 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.620973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70bedd5b-7206-41fe-b0a0-f19456abe21f-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-18035-predictor-5fdc678c5-z569k\" (UID: \"70bedd5b-7206-41fe-b0a0-f19456abe21f\") " pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:41:32.621570 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.621549    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70bedd5b-7206-41fe-b0a0-f19456abe21f-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-18035-predictor-5fdc678c5-z569k\" (UID: \"70bedd5b-7206-41fe-b0a0-f19456abe21f\") " pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:41:32.623349 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.623331    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70bedd5b-7206-41fe-b0a0-f19456abe21f-proxy-tls\") pod \"message-dumper-raw-18035-predictor-5fdc678c5-z569k\" (UID: \"70bedd5b-7206-41fe-b0a0-f19456abe21f\") " pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:41:32.636062 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.636036    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fr8c\" (UniqueName: \"kubernetes.io/projected/70bedd5b-7206-41fe-b0a0-f19456abe21f-kube-api-access-9fr8c\") pod \"message-dumper-raw-18035-predictor-5fdc678c5-z569k\" (UID: \"70bedd5b-7206-41fe-b0a0-f19456abe21f\") " pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:41:32.643442 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.643415    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:32.814340 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.814264    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:41:32.943914 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.943875    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"]
Jul 28 01:41:32.948490 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:41:32.948460    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bedd5b_7206_41fe_b0a0_f19456abe21f.slice/crio-f0d9e524b63ff13a711ced9ceab6b54c9e70dd427035f0a66495186fe2d28682 WatchSource:0}: Error finding container f0d9e524b63ff13a711ced9ceab6b54c9e70dd427035f0a66495186fe2d28682: Status 404 returned error can't find the container with id f0d9e524b63ff13a711ced9ceab6b54c9e70dd427035f0a66495186fe2d28682
Jul 28 01:41:32.950224 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:32.950209    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:41:33.657025 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:33.656994    2576 generic.go:356] "Generic (PLEG): container finished" podID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerID="e57f350804f7c64d2f4b9c5f4aaca66e717a0e1d95a5512acc9eb818ca53c700" exitCode=2
Jul 28 01:41:33.657211 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:33.657060    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" event={"ID":"96b0f7bb-0b37-49df-8970-0e19faf64778","Type":"ContainerDied","Data":"e57f350804f7c64d2f4b9c5f4aaca66e717a0e1d95a5512acc9eb818ca53c700"}
Jul 28 01:41:33.659200 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:33.659173    2576 generic.go:356] "Generic (PLEG): container finished" podID="1915c51c-c201-4148-97fb-f5b2b703171b" containerID="7f7bf86ccffa31eafc64377bd6ea476b4663569c9cbce06c268a96c8e899c766" exitCode=2
Jul 28 01:41:33.659312 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:33.659242    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" event={"ID":"1915c51c-c201-4148-97fb-f5b2b703171b","Type":"ContainerDied","Data":"7f7bf86ccffa31eafc64377bd6ea476b4663569c9cbce06c268a96c8e899c766"}
Jul 28 01:41:33.660223 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:33.660205    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k" event={"ID":"70bedd5b-7206-41fe-b0a0-f19456abe21f","Type":"ContainerStarted","Data":"f0d9e524b63ff13a711ced9ceab6b54c9e70dd427035f0a66495186fe2d28682"}
Jul 28 01:41:34.666796 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:34.666692    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k" event={"ID":"70bedd5b-7206-41fe-b0a0-f19456abe21f","Type":"ContainerStarted","Data":"b90340accfab7bfcb5022a670e82022f9444ef1ff0bfa77fd3c3dc2d8e886c74"}
Jul 28 01:41:34.666796 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:34.666756    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k" event={"ID":"70bedd5b-7206-41fe-b0a0-f19456abe21f","Type":"ContainerStarted","Data":"94d5e6ff0a0d148a7b714d713214801bfdf986ca88865d6e09b292ef8ca25653"}
Jul 28 01:41:34.666796 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:34.666786    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:41:34.667273 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:34.666850    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:41:34.668262 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:34.668245    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:41:34.686104 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:34.686055    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k" podStartSLOduration=1.37216202 podStartE2EDuration="2.686039606s" totalImagesPullingTime="1.313877586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:41:34.684959525 +0000 UTC m=+924.855998562" watchObservedRunningTime="2026-07-28 01:41:34.686039606 +0000 UTC m=+924.857078643"
Jul 28 01:41:35.308802 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:35.308758    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 28 01:41:35.314429 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:35.314405    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 28 01:41:36.313431 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.313373    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 28 01:41:36.318019 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.317872    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 01:41:36.449652 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.449627    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:41:36.552113 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.552023    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1915c51c-c201-4148-97fb-f5b2b703171b-kube-rbac-proxy-sar-config\") pod \"1915c51c-c201-4148-97fb-f5b2b703171b\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") "
Jul 28 01:41:36.552113 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.552063    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1915c51c-c201-4148-97fb-f5b2b703171b-kserve-provision-location\") pod \"1915c51c-c201-4148-97fb-f5b2b703171b\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") "
Jul 28 01:41:36.552113 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.552107    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1915c51c-c201-4148-97fb-f5b2b703171b-proxy-tls\") pod \"1915c51c-c201-4148-97fb-f5b2b703171b\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") "
Jul 28 01:41:36.552337 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.552161    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1915c51c-c201-4148-97fb-f5b2b703171b-cabundle-cert\") pod \"1915c51c-c201-4148-97fb-f5b2b703171b\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") "
Jul 28 01:41:36.552337 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.552195    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bgrt\" (UniqueName: \"kubernetes.io/projected/1915c51c-c201-4148-97fb-f5b2b703171b-kube-api-access-9bgrt\") pod \"1915c51c-c201-4148-97fb-f5b2b703171b\" (UID: \"1915c51c-c201-4148-97fb-f5b2b703171b\") "
Jul 28 01:41:36.552473 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.552452    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1915c51c-c201-4148-97fb-f5b2b703171b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1915c51c-c201-4148-97fb-f5b2b703171b" (UID: "1915c51c-c201-4148-97fb-f5b2b703171b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:36.552534 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.552482    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1915c51c-c201-4148-97fb-f5b2b703171b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1915c51c-c201-4148-97fb-f5b2b703171b" (UID: "1915c51c-c201-4148-97fb-f5b2b703171b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:36.552585 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.552530    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1915c51c-c201-4148-97fb-f5b2b703171b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1915c51c-c201-4148-97fb-f5b2b703171b" (UID: "1915c51c-c201-4148-97fb-f5b2b703171b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:36.554412 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.554371    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1915c51c-c201-4148-97fb-f5b2b703171b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1915c51c-c201-4148-97fb-f5b2b703171b" (UID: "1915c51c-c201-4148-97fb-f5b2b703171b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:41:36.554537 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.554424    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1915c51c-c201-4148-97fb-f5b2b703171b-kube-api-access-9bgrt" (OuterVolumeSpecName: "kube-api-access-9bgrt") pod "1915c51c-c201-4148-97fb-f5b2b703171b" (UID: "1915c51c-c201-4148-97fb-f5b2b703171b"). InnerVolumeSpecName "kube-api-access-9bgrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:36.652959 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.652921    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bgrt\" (UniqueName: \"kubernetes.io/projected/1915c51c-c201-4148-97fb-f5b2b703171b-kube-api-access-9bgrt\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:41:36.652959 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.652953    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1915c51c-c201-4148-97fb-f5b2b703171b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:41:36.652959 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.652963    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1915c51c-c201-4148-97fb-f5b2b703171b-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:41:36.652959 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.652972    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1915c51c-c201-4148-97fb-f5b2b703171b-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:41:36.653225 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.652982    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1915c51c-c201-4148-97fb-f5b2b703171b-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:41:36.674545 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.674515    2576 generic.go:356] "Generic (PLEG): container finished" podID="1915c51c-c201-4148-97fb-f5b2b703171b" containerID="48c41ec4f0abad4a09f369c3eb561dbe9e9d4069ec85b7f0f0b007584c1d6c28" exitCode=0
Jul 28 01:41:36.674696 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.674592    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"
Jul 28 01:41:36.674696 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.674611    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" event={"ID":"1915c51c-c201-4148-97fb-f5b2b703171b","Type":"ContainerDied","Data":"48c41ec4f0abad4a09f369c3eb561dbe9e9d4069ec85b7f0f0b007584c1d6c28"}
Jul 28 01:41:36.674773 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.674701    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4" event={"ID":"1915c51c-c201-4148-97fb-f5b2b703171b","Type":"ContainerDied","Data":"324cf382749563a926b8ee1c781b946f73b4141f0c16223920e6ad8cee50190c"}
Jul 28 01:41:36.674773 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.674730    2576 scope.go:117] "RemoveContainer" containerID="7f7bf86ccffa31eafc64377bd6ea476b4663569c9cbce06c268a96c8e899c766"
Jul 28 01:41:36.683851 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.683831    2576 scope.go:117] "RemoveContainer" containerID="48c41ec4f0abad4a09f369c3eb561dbe9e9d4069ec85b7f0f0b007584c1d6c28"
Jul 28 01:41:36.690930 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.690912    2576 scope.go:117] "RemoveContainer" containerID="2151539968d438796f2a6e24a5c55348edfb31fea907a9d3a28da5bfdc143e1f"
Jul 28 01:41:36.696164 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.696139    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"]
Jul 28 01:41:36.697886 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.697871    2576 scope.go:117] "RemoveContainer" containerID="7f7bf86ccffa31eafc64377bd6ea476b4663569c9cbce06c268a96c8e899c766"
Jul 28 01:41:36.698211 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:41:36.698125    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f7bf86ccffa31eafc64377bd6ea476b4663569c9cbce06c268a96c8e899c766\": container with ID starting with 7f7bf86ccffa31eafc64377bd6ea476b4663569c9cbce06c268a96c8e899c766 not found: ID does not exist" containerID="7f7bf86ccffa31eafc64377bd6ea476b4663569c9cbce06c268a96c8e899c766"
Jul 28 01:41:36.698290 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.698222    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f7bf86ccffa31eafc64377bd6ea476b4663569c9cbce06c268a96c8e899c766"} err="failed to get container status \"7f7bf86ccffa31eafc64377bd6ea476b4663569c9cbce06c268a96c8e899c766\": rpc error: code = NotFound desc = could not find container \"7f7bf86ccffa31eafc64377bd6ea476b4663569c9cbce06c268a96c8e899c766\": container with ID starting with 7f7bf86ccffa31eafc64377bd6ea476b4663569c9cbce06c268a96c8e899c766 not found: ID does not exist"
Jul 28 01:41:36.698290 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.698244    2576 scope.go:117] "RemoveContainer" containerID="48c41ec4f0abad4a09f369c3eb561dbe9e9d4069ec85b7f0f0b007584c1d6c28"
Jul 28 01:41:36.699093 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:41:36.699055    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c41ec4f0abad4a09f369c3eb561dbe9e9d4069ec85b7f0f0b007584c1d6c28\": container with ID starting with 48c41ec4f0abad4a09f369c3eb561dbe9e9d4069ec85b7f0f0b007584c1d6c28 not found: ID does not exist" containerID="48c41ec4f0abad4a09f369c3eb561dbe9e9d4069ec85b7f0f0b007584c1d6c28"
Jul 28 01:41:36.699093 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.699086    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c41ec4f0abad4a09f369c3eb561dbe9e9d4069ec85b7f0f0b007584c1d6c28"} err="failed to get container status \"48c41ec4f0abad4a09f369c3eb561dbe9e9d4069ec85b7f0f0b007584c1d6c28\": rpc error: code = NotFound desc = could not find container \"48c41ec4f0abad4a09f369c3eb561dbe9e9d4069ec85b7f0f0b007584c1d6c28\": container with ID starting with 48c41ec4f0abad4a09f369c3eb561dbe9e9d4069ec85b7f0f0b007584c1d6c28 not found: ID does not exist"
Jul 28 01:41:36.699692 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.699106    2576 scope.go:117] "RemoveContainer" containerID="2151539968d438796f2a6e24a5c55348edfb31fea907a9d3a28da5bfdc143e1f"
Jul 28 01:41:36.699800 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:41:36.699778    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2151539968d438796f2a6e24a5c55348edfb31fea907a9d3a28da5bfdc143e1f\": container with ID starting with 2151539968d438796f2a6e24a5c55348edfb31fea907a9d3a28da5bfdc143e1f not found: ID does not exist" containerID="2151539968d438796f2a6e24a5c55348edfb31fea907a9d3a28da5bfdc143e1f"
Jul 28 01:41:36.699857 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.699807    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2151539968d438796f2a6e24a5c55348edfb31fea907a9d3a28da5bfdc143e1f"} err="failed to get container status \"2151539968d438796f2a6e24a5c55348edfb31fea907a9d3a28da5bfdc143e1f\": rpc error: code = NotFound desc = could not find container \"2151539968d438796f2a6e24a5c55348edfb31fea907a9d3a28da5bfdc143e1f\": container with ID starting with 2151539968d438796f2a6e24a5c55348edfb31fea907a9d3a28da5bfdc143e1f not found: ID does not exist"
Jul 28 01:41:36.700158 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:36.700142    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cd9f7-predictor-69c7b46449-8fqj4"]
Jul 28 01:41:37.506839 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.506815    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:41:37.558548 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.558471    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96b0f7bb-0b37-49df-8970-0e19faf64778-proxy-tls\") pod \"96b0f7bb-0b37-49df-8970-0e19faf64778\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") "
Jul 28 01:41:37.558548 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.558512    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9btb\" (UniqueName: \"kubernetes.io/projected/96b0f7bb-0b37-49df-8970-0e19faf64778-kube-api-access-z9btb\") pod \"96b0f7bb-0b37-49df-8970-0e19faf64778\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") "
Jul 28 01:41:37.558757 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.558552    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96b0f7bb-0b37-49df-8970-0e19faf64778-kube-rbac-proxy-sar-config\") pod \"96b0f7bb-0b37-49df-8970-0e19faf64778\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") "
Jul 28 01:41:37.558757 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.558580    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96b0f7bb-0b37-49df-8970-0e19faf64778-kserve-provision-location\") pod \"96b0f7bb-0b37-49df-8970-0e19faf64778\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") "
Jul 28 01:41:37.558757 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.558603    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96b0f7bb-0b37-49df-8970-0e19faf64778-cabundle-cert\") pod \"96b0f7bb-0b37-49df-8970-0e19faf64778\" (UID: \"96b0f7bb-0b37-49df-8970-0e19faf64778\") "
Jul 28 01:41:37.558948 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.558922    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b0f7bb-0b37-49df-8970-0e19faf64778-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96b0f7bb-0b37-49df-8970-0e19faf64778" (UID: "96b0f7bb-0b37-49df-8970-0e19faf64778"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:37.559002 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.558975    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b0f7bb-0b37-49df-8970-0e19faf64778-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "96b0f7bb-0b37-49df-8970-0e19faf64778" (UID: "96b0f7bb-0b37-49df-8970-0e19faf64778"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:37.559037 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.559009    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b0f7bb-0b37-49df-8970-0e19faf64778-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "96b0f7bb-0b37-49df-8970-0e19faf64778" (UID: "96b0f7bb-0b37-49df-8970-0e19faf64778"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:37.560698 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.560679    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b0f7bb-0b37-49df-8970-0e19faf64778-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "96b0f7bb-0b37-49df-8970-0e19faf64778" (UID: "96b0f7bb-0b37-49df-8970-0e19faf64778"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:41:37.560775 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.560754    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b0f7bb-0b37-49df-8970-0e19faf64778-kube-api-access-z9btb" (OuterVolumeSpecName: "kube-api-access-z9btb") pod "96b0f7bb-0b37-49df-8970-0e19faf64778" (UID: "96b0f7bb-0b37-49df-8970-0e19faf64778"). InnerVolumeSpecName "kube-api-access-z9btb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:37.659946 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.659905    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96b0f7bb-0b37-49df-8970-0e19faf64778-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:41:37.659946 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.659937    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9btb\" (UniqueName: \"kubernetes.io/projected/96b0f7bb-0b37-49df-8970-0e19faf64778-kube-api-access-z9btb\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:41:37.659946 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.659949    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96b0f7bb-0b37-49df-8970-0e19faf64778-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:41:37.659946 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.659958    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96b0f7bb-0b37-49df-8970-0e19faf64778-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:41:37.660200 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.659967    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96b0f7bb-0b37-49df-8970-0e19faf64778-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:41:37.680435 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.680398    2576 generic.go:356] "Generic (PLEG): container finished" podID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerID="5b365c26a86bbc1c5ca0a364ee3b05aaa89fdf07c034a497879b6281ea076ebd" exitCode=0
Jul 28 01:41:37.680435 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.680435    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" event={"ID":"96b0f7bb-0b37-49df-8970-0e19faf64778","Type":"ContainerDied","Data":"5b365c26a86bbc1c5ca0a364ee3b05aaa89fdf07c034a497879b6281ea076ebd"}
Jul 28 01:41:37.680633 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.680458    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"
Jul 28 01:41:37.680633 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.680490    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn" event={"ID":"96b0f7bb-0b37-49df-8970-0e19faf64778","Type":"ContainerDied","Data":"3843e083363040b7599fbc64d66b80fa9a03a7e2d361485091b06ccfa183028b"}
Jul 28 01:41:37.680633 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.680518    2576 scope.go:117] "RemoveContainer" containerID="e57f350804f7c64d2f4b9c5f4aaca66e717a0e1d95a5512acc9eb818ca53c700"
Jul 28 01:41:37.690807 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.690788    2576 scope.go:117] "RemoveContainer" containerID="5b365c26a86bbc1c5ca0a364ee3b05aaa89fdf07c034a497879b6281ea076ebd"
Jul 28 01:41:37.697900 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.697879    2576 scope.go:117] "RemoveContainer" containerID="2356ffb0209706d16538c6ca969527c6b7c4f3c694a0a66327589c2bd66db90d"
Jul 28 01:41:37.703879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.703859    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"]
Jul 28 01:41:37.704966 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.704950    2576 scope.go:117] "RemoveContainer" containerID="e57f350804f7c64d2f4b9c5f4aaca66e717a0e1d95a5512acc9eb818ca53c700"
Jul 28 01:41:37.705229 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:41:37.705212    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57f350804f7c64d2f4b9c5f4aaca66e717a0e1d95a5512acc9eb818ca53c700\": container with ID starting with e57f350804f7c64d2f4b9c5f4aaca66e717a0e1d95a5512acc9eb818ca53c700 not found: ID does not exist" containerID="e57f350804f7c64d2f4b9c5f4aaca66e717a0e1d95a5512acc9eb818ca53c700"
Jul 28 01:41:37.705275 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.705236    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57f350804f7c64d2f4b9c5f4aaca66e717a0e1d95a5512acc9eb818ca53c700"} err="failed to get container status \"e57f350804f7c64d2f4b9c5f4aaca66e717a0e1d95a5512acc9eb818ca53c700\": rpc error: code = NotFound desc = could not find container \"e57f350804f7c64d2f4b9c5f4aaca66e717a0e1d95a5512acc9eb818ca53c700\": container with ID starting with e57f350804f7c64d2f4b9c5f4aaca66e717a0e1d95a5512acc9eb818ca53c700 not found: ID does not exist"
Jul 28 01:41:37.705275 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.705251    2576 scope.go:117] "RemoveContainer" containerID="5b365c26a86bbc1c5ca0a364ee3b05aaa89fdf07c034a497879b6281ea076ebd"
Jul 28 01:41:37.705482 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:41:37.705466    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b365c26a86bbc1c5ca0a364ee3b05aaa89fdf07c034a497879b6281ea076ebd\": container with ID starting with 5b365c26a86bbc1c5ca0a364ee3b05aaa89fdf07c034a497879b6281ea076ebd not found: ID does not exist" containerID="5b365c26a86bbc1c5ca0a364ee3b05aaa89fdf07c034a497879b6281ea076ebd"
Jul 28 01:41:37.705525 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.705486    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b365c26a86bbc1c5ca0a364ee3b05aaa89fdf07c034a497879b6281ea076ebd"} err="failed to get container status \"5b365c26a86bbc1c5ca0a364ee3b05aaa89fdf07c034a497879b6281ea076ebd\": rpc error: code = NotFound desc = could not find container \"5b365c26a86bbc1c5ca0a364ee3b05aaa89fdf07c034a497879b6281ea076ebd\": container with ID starting with 5b365c26a86bbc1c5ca0a364ee3b05aaa89fdf07c034a497879b6281ea076ebd not found: ID does not exist"
Jul 28 01:41:37.705525 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.705499    2576 scope.go:117] "RemoveContainer" containerID="2356ffb0209706d16538c6ca969527c6b7c4f3c694a0a66327589c2bd66db90d"
Jul 28 01:41:37.705682 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:41:37.705668    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2356ffb0209706d16538c6ca969527c6b7c4f3c694a0a66327589c2bd66db90d\": container with ID starting with 2356ffb0209706d16538c6ca969527c6b7c4f3c694a0a66327589c2bd66db90d not found: ID does not exist" containerID="2356ffb0209706d16538c6ca969527c6b7c4f3c694a0a66327589c2bd66db90d"
Jul 28 01:41:37.705720 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.705685    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2356ffb0209706d16538c6ca969527c6b7c4f3c694a0a66327589c2bd66db90d"} err="failed to get container status \"2356ffb0209706d16538c6ca969527c6b7c4f3c694a0a66327589c2bd66db90d\": rpc error: code = NotFound desc = could not find container \"2356ffb0209706d16538c6ca969527c6b7c4f3c694a0a66327589c2bd66db90d\": container with ID starting with 2356ffb0209706d16538c6ca969527c6b7c4f3c694a0a66327589c2bd66db90d not found: ID does not exist"
Jul 28 01:41:37.708059 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:37.708038    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cd9f7-predictor-58b8468b88-bkpmn"]
Jul 28 01:41:38.431397 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:38.431357    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" path="/var/lib/kubelet/pods/1915c51c-c201-4148-97fb-f5b2b703171b/volumes"
Jul 28 01:41:38.431842 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:38.431828    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" path="/var/lib/kubelet/pods/96b0f7bb-0b37-49df-8970-0e19faf64778/volumes"
Jul 28 01:41:41.680044 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:41.680017    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:41:42.497029 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.496998    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:42.554720 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.554683    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"]
Jul 28 01:41:42.554934 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.554922    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kserve-container"
Jul 28 01:41:42.554976 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.554936    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kserve-container"
Jul 28 01:41:42.554976 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.554947    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="storage-initializer"
Jul 28 01:41:42.554976 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.554952    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="storage-initializer"
Jul 28 01:41:42.554976 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.554964    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="storage-initializer"
Jul 28 01:41:42.554976 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.554970    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="storage-initializer"
Jul 28 01:41:42.554976 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.554976    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kube-rbac-proxy"
Jul 28 01:41:42.555144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.554981    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kube-rbac-proxy"
Jul 28 01:41:42.555144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.554988    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kserve-container"
Jul 28 01:41:42.555144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.554993    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kserve-container"
Jul 28 01:41:42.555144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.554999    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kube-rbac-proxy"
Jul 28 01:41:42.555144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.555004    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kube-rbac-proxy"
Jul 28 01:41:42.555144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.555046    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kserve-container"
Jul 28 01:41:42.555144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.555052    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kube-rbac-proxy"
Jul 28 01:41:42.555144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.555059    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="96b0f7bb-0b37-49df-8970-0e19faf64778" containerName="kube-rbac-proxy"
Jul 28 01:41:42.555144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.555065    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1915c51c-c201-4148-97fb-f5b2b703171b" containerName="kserve-container"
Jul 28 01:41:42.559769 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.559748    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.562054 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.562029    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-18035-predictor-serving-cert\""
Jul 28 01:41:42.562210 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.562062    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 01:41:42.562210 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.562084    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 01:41:42.562405 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.562371    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-18035-kube-rbac-proxy-sar-config\""
Jul 28 01:41:42.575424 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.575371    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"]
Jul 28 01:41:42.594425 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.594393    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/186f4dca-65e7-49c8-8476-21ebd5bca49d-proxy-tls\") pod \"isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.594560 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.594462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/186f4dca-65e7-49c8-8476-21ebd5bca49d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.594560 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.594496    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/186f4dca-65e7-49c8-8476-21ebd5bca49d-cabundle-cert\") pod \"isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.594560 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.594549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qktbw\" (UniqueName: \"kubernetes.io/projected/186f4dca-65e7-49c8-8476-21ebd5bca49d-kube-api-access-qktbw\") pod \"isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.594656 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.594575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/186f4dca-65e7-49c8-8476-21ebd5bca49d-kserve-provision-location\") pod \"isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.595894 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.595875    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:42.694882 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.694849    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/186f4dca-65e7-49c8-8476-21ebd5bca49d-proxy-tls\") pod \"isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.694882 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.694888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/186f4dca-65e7-49c8-8476-21ebd5bca49d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.695430 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.694911    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/186f4dca-65e7-49c8-8476-21ebd5bca49d-cabundle-cert\") pod \"isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.695430 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.694933    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qktbw\" (UniqueName: \"kubernetes.io/projected/186f4dca-65e7-49c8-8476-21ebd5bca49d-kube-api-access-qktbw\") pod \"isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.695430 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.694956    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/186f4dca-65e7-49c8-8476-21ebd5bca49d-kserve-provision-location\") pod \"isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.695618 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.695442    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/186f4dca-65e7-49c8-8476-21ebd5bca49d-kserve-provision-location\") pod \"isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.695672 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.695630    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/186f4dca-65e7-49c8-8476-21ebd5bca49d-cabundle-cert\") pod \"isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.695672 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.695653    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/186f4dca-65e7-49c8-8476-21ebd5bca49d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.697742 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.697723    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/186f4dca-65e7-49c8-8476-21ebd5bca49d-proxy-tls\") pod \"isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.703328 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.703300    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qktbw\" (UniqueName: \"kubernetes.io/projected/186f4dca-65e7-49c8-8476-21ebd5bca49d-kube-api-access-qktbw\") pod \"isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:42.870990 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:42.870905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:43.004477 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:43.004446    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"]
Jul 28 01:41:43.007400 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:41:43.007350    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186f4dca_65e7_49c8_8476_21ebd5bca49d.slice/crio-09fb271dc5049d7f3c030fb8976193e6304d4599356fed644b9457a6a056fc25 WatchSource:0}: Error finding container 09fb271dc5049d7f3c030fb8976193e6304d4599356fed644b9457a6a056fc25: Status 404 returned error can't find the container with id 09fb271dc5049d7f3c030fb8976193e6304d4599356fed644b9457a6a056fc25
Jul 28 01:41:43.702753 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:43.702709    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" event={"ID":"186f4dca-65e7-49c8-8476-21ebd5bca49d","Type":"ContainerStarted","Data":"764c4074e5cc55e897cf640094264af113f4c72db70757cd5389361d4928253a"}
Jul 28 01:41:43.703240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:43.702784    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" event={"ID":"186f4dca-65e7-49c8-8476-21ebd5bca49d","Type":"ContainerStarted","Data":"09fb271dc5049d7f3c030fb8976193e6304d4599356fed644b9457a6a056fc25"}
Jul 28 01:41:47.718141 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:47.718104    2576 generic.go:356] "Generic (PLEG): container finished" podID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerID="764c4074e5cc55e897cf640094264af113f4c72db70757cd5389361d4928253a" exitCode=0
Jul 28 01:41:47.718757 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:47.718193    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" event={"ID":"186f4dca-65e7-49c8-8476-21ebd5bca49d","Type":"ContainerDied","Data":"764c4074e5cc55e897cf640094264af113f4c72db70757cd5389361d4928253a"}
Jul 28 01:41:48.724582 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:48.724547    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" event={"ID":"186f4dca-65e7-49c8-8476-21ebd5bca49d","Type":"ContainerStarted","Data":"7e6114d8523e284743dac4fb727a163fc533a9486811fc4208d94fc21e09859a"}
Jul 28 01:41:48.725066 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:48.724612    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" event={"ID":"186f4dca-65e7-49c8-8476-21ebd5bca49d","Type":"ContainerStarted","Data":"31fdf407e046931459f5a5c86fed88ffccd386c6824564fd067634d4363b4442"}
Jul 28 01:41:48.725066 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:48.724636    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" event={"ID":"186f4dca-65e7-49c8-8476-21ebd5bca49d","Type":"ContainerStarted","Data":"12cf1c5eede18e96250790bb5f571a9a65b95ac345d77f16bb7160829f4a38f0"}
Jul 28 01:41:48.725066 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:48.724924    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:48.725066 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:48.725003    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:48.725066 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:48.725033    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:48.726504 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:48.726476    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:41:48.727209 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:48.727186    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:41:48.749093 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:48.749050    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podStartSLOduration=2.892235658 podStartE2EDuration="6.749036951s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.856801293s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:41:48.74870805 +0000 UTC m=+938.919747099" watchObservedRunningTime="2026-07-28 01:41:48.749036951 +0000 UTC m=+938.920076032"
Jul 28 01:41:49.728544 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:49.728498    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:41:49.728959 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:49.728934    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:41:54.733378 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:54.733347    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:41:54.734003 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:54.733977    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:41:54.734228 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:41:54.734205    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:42:04.734207 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:42:04.734149    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:42:04.734686 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:42:04.734628    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:42:14.734298 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:42:14.734254    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:42:14.734741 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:42:14.734717    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:42:24.734628 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:42:24.734583    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:42:24.735146 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:42:24.734986    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:42:34.734497 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:42:34.734375    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:42:34.734907 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:42:34.734847    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:42:44.734453 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:42:44.734407    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:42:44.734870 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:42:44.734842    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:42:54.734287 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:42:54.734232    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:42:54.734723 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:42:54.734632    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:43:04.734582 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:04.734545    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:43:04.735091 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:04.735069    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:43:17.494923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.494895    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-18035-predictor-5fdc678c5-z569k_70bedd5b-7206-41fe-b0a0-f19456abe21f/kserve-container/0.log"
Jul 28 01:43:17.721312 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.721282    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"]
Jul 28 01:43:17.721640 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.721589    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container" containerID="cri-o://12cf1c5eede18e96250790bb5f571a9a65b95ac345d77f16bb7160829f4a38f0" gracePeriod=30
Jul 28 01:43:17.721774 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.721639    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kube-rbac-proxy" containerID="cri-o://31fdf407e046931459f5a5c86fed88ffccd386c6824564fd067634d4363b4442" gracePeriod=30
Jul 28 01:43:17.721774 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.721628    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent" containerID="cri-o://7e6114d8523e284743dac4fb727a163fc533a9486811fc4208d94fc21e09859a" gracePeriod=30
Jul 28 01:43:17.757662 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.757588    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:43:17.797032 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.796976    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"]
Jul 28 01:43:17.800278 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.800260    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:17.802480 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.802455    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-6ab94-kube-rbac-proxy-sar-config\""
Jul 28 01:43:17.802480 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.802477    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-6ab94-predictor-serving-cert\""
Jul 28 01:43:17.813065 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.813033    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"]
Jul 28 01:43:17.839776 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.839748    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"]
Jul 28 01:43:17.840000 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.839972    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k" podUID="70bedd5b-7206-41fe-b0a0-f19456abe21f" containerName="kserve-container" containerID="cri-o://94d5e6ff0a0d148a7b714d713214801bfdf986ca88865d6e09b292ef8ca25653" gracePeriod=30
Jul 28 01:43:17.840118 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.839998    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k" podUID="70bedd5b-7206-41fe-b0a0-f19456abe21f" containerName="kube-rbac-proxy" containerID="cri-o://b90340accfab7bfcb5022a670e82022f9444ef1ff0bfa77fd3c3dc2d8e886c74" gracePeriod=30
Jul 28 01:43:17.875252 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.875229    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/104ebc58-a752-4fdc-bea2-30aec6ddb112-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:17.875366 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.875273    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104ebc58-a752-4fdc-bea2-30aec6ddb112-proxy-tls\") pod \"isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:17.875366 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.875311    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104ebc58-a752-4fdc-bea2-30aec6ddb112-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:17.875480 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.875425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t57pb\" (UniqueName: \"kubernetes.io/projected/104ebc58-a752-4fdc-bea2-30aec6ddb112-kube-api-access-t57pb\") pod \"isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:17.875519 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.875474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/104ebc58-a752-4fdc-bea2-30aec6ddb112-cabundle-cert\") pod \"isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:17.975907 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.975872    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/104ebc58-a752-4fdc-bea2-30aec6ddb112-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:17.976081 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.975932    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104ebc58-a752-4fdc-bea2-30aec6ddb112-proxy-tls\") pod \"isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:17.976081 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.975979    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104ebc58-a752-4fdc-bea2-30aec6ddb112-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:17.976081 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.976011    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t57pb\" (UniqueName: \"kubernetes.io/projected/104ebc58-a752-4fdc-bea2-30aec6ddb112-kube-api-access-t57pb\") pod \"isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:17.976081 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.976047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/104ebc58-a752-4fdc-bea2-30aec6ddb112-cabundle-cert\") pod \"isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:17.976310 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.976286    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/104ebc58-a752-4fdc-bea2-30aec6ddb112-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:17.976899 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.976875    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104ebc58-a752-4fdc-bea2-30aec6ddb112-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:17.977123 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.977102    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/104ebc58-a752-4fdc-bea2-30aec6ddb112-cabundle-cert\") pod \"isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:17.978590 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.978570    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104ebc58-a752-4fdc-bea2-30aec6ddb112-proxy-tls\") pod \"isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:17.983721 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:17.983697    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t57pb\" (UniqueName: \"kubernetes.io/projected/104ebc58-a752-4fdc-bea2-30aec6ddb112-kube-api-access-t57pb\") pod \"isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:18.037932 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.037904    2576 generic.go:356] "Generic (PLEG): container finished" podID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerID="31fdf407e046931459f5a5c86fed88ffccd386c6824564fd067634d4363b4442" exitCode=2
Jul 28 01:43:18.038069 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.037972    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" event={"ID":"186f4dca-65e7-49c8-8476-21ebd5bca49d","Type":"ContainerDied","Data":"31fdf407e046931459f5a5c86fed88ffccd386c6824564fd067634d4363b4442"}
Jul 28 01:43:18.039948 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.039927    2576 generic.go:356] "Generic (PLEG): container finished" podID="70bedd5b-7206-41fe-b0a0-f19456abe21f" containerID="b90340accfab7bfcb5022a670e82022f9444ef1ff0bfa77fd3c3dc2d8e886c74" exitCode=2
Jul 28 01:43:18.039948 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.039944    2576 generic.go:356] "Generic (PLEG): container finished" podID="70bedd5b-7206-41fe-b0a0-f19456abe21f" containerID="94d5e6ff0a0d148a7b714d713214801bfdf986ca88865d6e09b292ef8ca25653" exitCode=2
Jul 28 01:43:18.040111 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.039975    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k" event={"ID":"70bedd5b-7206-41fe-b0a0-f19456abe21f","Type":"ContainerDied","Data":"b90340accfab7bfcb5022a670e82022f9444ef1ff0bfa77fd3c3dc2d8e886c74"}
Jul 28 01:43:18.040111 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.040021    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k" event={"ID":"70bedd5b-7206-41fe-b0a0-f19456abe21f","Type":"ContainerDied","Data":"94d5e6ff0a0d148a7b714d713214801bfdf986ca88865d6e09b292ef8ca25653"}
Jul 28 01:43:18.082650 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.082628    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:43:18.110373 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.110343    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:18.177470 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.177429    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fr8c\" (UniqueName: \"kubernetes.io/projected/70bedd5b-7206-41fe-b0a0-f19456abe21f-kube-api-access-9fr8c\") pod \"70bedd5b-7206-41fe-b0a0-f19456abe21f\" (UID: \"70bedd5b-7206-41fe-b0a0-f19456abe21f\") "
Jul 28 01:43:18.177612 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.177514    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70bedd5b-7206-41fe-b0a0-f19456abe21f-proxy-tls\") pod \"70bedd5b-7206-41fe-b0a0-f19456abe21f\" (UID: \"70bedd5b-7206-41fe-b0a0-f19456abe21f\") "
Jul 28 01:43:18.177612 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.177549    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70bedd5b-7206-41fe-b0a0-f19456abe21f-kube-rbac-proxy-sar-config\") pod \"70bedd5b-7206-41fe-b0a0-f19456abe21f\" (UID: \"70bedd5b-7206-41fe-b0a0-f19456abe21f\") "
Jul 28 01:43:18.177933 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.177909    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bedd5b-7206-41fe-b0a0-f19456abe21f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "70bedd5b-7206-41fe-b0a0-f19456abe21f" (UID: "70bedd5b-7206-41fe-b0a0-f19456abe21f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:43:18.180190 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.180163    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70bedd5b-7206-41fe-b0a0-f19456abe21f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70bedd5b-7206-41fe-b0a0-f19456abe21f" (UID: "70bedd5b-7206-41fe-b0a0-f19456abe21f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:43:18.180190 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.180177    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bedd5b-7206-41fe-b0a0-f19456abe21f-kube-api-access-9fr8c" (OuterVolumeSpecName: "kube-api-access-9fr8c") pod "70bedd5b-7206-41fe-b0a0-f19456abe21f" (UID: "70bedd5b-7206-41fe-b0a0-f19456abe21f"). InnerVolumeSpecName "kube-api-access-9fr8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:43:18.240098 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.240074    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"]
Jul 28 01:43:18.242544 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:43:18.242514    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104ebc58_a752_4fdc_bea2_30aec6ddb112.slice/crio-054f0fc4311dd1e525a7bab7eda35a8a880c1ecc31b21c0970aa682ee9dd4711 WatchSource:0}: Error finding container 054f0fc4311dd1e525a7bab7eda35a8a880c1ecc31b21c0970aa682ee9dd4711: Status 404 returned error can't find the container with id 054f0fc4311dd1e525a7bab7eda35a8a880c1ecc31b21c0970aa682ee9dd4711
Jul 28 01:43:18.278166 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.278141    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70bedd5b-7206-41fe-b0a0-f19456abe21f-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:43:18.278261 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.278189    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70bedd5b-7206-41fe-b0a0-f19456abe21f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:43:18.278261 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:18.278205    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fr8c\" (UniqueName: \"kubernetes.io/projected/70bedd5b-7206-41fe-b0a0-f19456abe21f-kube-api-access-9fr8c\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:43:19.038992 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:19.038909    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:43:19.045163 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:19.045142    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"
Jul 28 01:43:19.045321 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:19.045141    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k" event={"ID":"70bedd5b-7206-41fe-b0a0-f19456abe21f","Type":"ContainerDied","Data":"f0d9e524b63ff13a711ced9ceab6b54c9e70dd427035f0a66495186fe2d28682"}
Jul 28 01:43:19.045321 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:19.045275    2576 scope.go:117] "RemoveContainer" containerID="b90340accfab7bfcb5022a670e82022f9444ef1ff0bfa77fd3c3dc2d8e886c74"
Jul 28 01:43:19.046801 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:19.046768    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" event={"ID":"104ebc58-a752-4fdc-bea2-30aec6ddb112","Type":"ContainerStarted","Data":"1fd4898fb1beb1daf63fbf7bb84c93e67f1e844c5df939c198550673350155e1"}
Jul 28 01:43:19.046904 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:19.046820    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" event={"ID":"104ebc58-a752-4fdc-bea2-30aec6ddb112","Type":"ContainerStarted","Data":"054f0fc4311dd1e525a7bab7eda35a8a880c1ecc31b21c0970aa682ee9dd4711"}
Jul 28 01:43:19.053133 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:19.053112    2576 scope.go:117] "RemoveContainer" containerID="94d5e6ff0a0d148a7b714d713214801bfdf986ca88865d6e09b292ef8ca25653"
Jul 28 01:43:19.086544 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:19.086513    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"]
Jul 28 01:43:19.090595 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:19.090570    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-18035-predictor-5fdc678c5-z569k"]
Jul 28 01:43:19.729444 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:19.729406    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 28 01:43:20.430718 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:20.430687    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70bedd5b-7206-41fe-b0a0-f19456abe21f" path="/var/lib/kubelet/pods/70bedd5b-7206-41fe-b0a0-f19456abe21f/volumes"
Jul 28 01:43:22.060993 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:22.060962    2576 generic.go:356] "Generic (PLEG): container finished" podID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerID="1fd4898fb1beb1daf63fbf7bb84c93e67f1e844c5df939c198550673350155e1" exitCode=0
Jul 28 01:43:22.061363 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:22.061051    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" event={"ID":"104ebc58-a752-4fdc-bea2-30aec6ddb112","Type":"ContainerDied","Data":"1fd4898fb1beb1daf63fbf7bb84c93e67f1e844c5df939c198550673350155e1"}
Jul 28 01:43:23.068120 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:23.068088    2576 generic.go:356] "Generic (PLEG): container finished" podID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerID="12cf1c5eede18e96250790bb5f571a9a65b95ac345d77f16bb7160829f4a38f0" exitCode=0
Jul 28 01:43:23.068541 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:23.068160    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" event={"ID":"186f4dca-65e7-49c8-8476-21ebd5bca49d","Type":"ContainerDied","Data":"12cf1c5eede18e96250790bb5f571a9a65b95ac345d77f16bb7160829f4a38f0"}
Jul 28 01:43:23.070225 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:23.070201    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" event={"ID":"104ebc58-a752-4fdc-bea2-30aec6ddb112","Type":"ContainerStarted","Data":"c1536f272c51f80a955a0718ff0d0ddc2b0ec3e70b695b2d7301aa40dffcf6ae"}
Jul 28 01:43:23.070366 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:23.070243    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" event={"ID":"104ebc58-a752-4fdc-bea2-30aec6ddb112","Type":"ContainerStarted","Data":"790310339e371cfb71f39605a25a58b5ebfa2a3d54505f9c9c3608ce843b514c"}
Jul 28 01:43:23.070512 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:23.070497    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:23.070675 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:23.070651    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:23.071807 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:23.071783    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:43:23.110569 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:23.110520    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podStartSLOduration=2.399383168 podStartE2EDuration="6.110504325s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.711121157s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:43:23.109869025 +0000 UTC m=+1033.280908059" watchObservedRunningTime="2026-07-28 01:43:23.110504325 +0000 UTC m=+1033.281543387"
Jul 28 01:43:24.074398 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:24.074354    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:43:24.728980 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:24.728940    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 28 01:43:24.734339 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:24.734293    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:43:24.735340 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:24.735320    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:43:29.079341 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:29.079313    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:43:29.079961 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:29.079931    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:43:29.729410 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:29.729350    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 28 01:43:29.729602 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:29.729540    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:43:34.729558 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:34.729515    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 28 01:43:34.733875 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:34.733853    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:43:34.735481 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:34.735446    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:43:39.080593 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:39.080549    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:43:39.728793 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:39.728748    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 28 01:43:44.174964 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:44.174914    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:43:44.237342 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:44.237298    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:43:44.729551 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:44.729512    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 28 01:43:44.733848 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:44.733828    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:43:44.733960 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:44.733944    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:43:44.735430 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:44.735406    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:43:44.735508 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:44.735502    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:43:47.873240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:47.873217    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:43:47.999810 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:47.999782    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/186f4dca-65e7-49c8-8476-21ebd5bca49d-proxy-tls\") pod \"186f4dca-65e7-49c8-8476-21ebd5bca49d\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") "
Jul 28 01:43:47.999970 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:47.999833    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/186f4dca-65e7-49c8-8476-21ebd5bca49d-cabundle-cert\") pod \"186f4dca-65e7-49c8-8476-21ebd5bca49d\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") "
Jul 28 01:43:47.999970 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:47.999861    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/186f4dca-65e7-49c8-8476-21ebd5bca49d-kserve-provision-location\") pod \"186f4dca-65e7-49c8-8476-21ebd5bca49d\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") "
Jul 28 01:43:47.999970 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:47.999883    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qktbw\" (UniqueName: \"kubernetes.io/projected/186f4dca-65e7-49c8-8476-21ebd5bca49d-kube-api-access-qktbw\") pod \"186f4dca-65e7-49c8-8476-21ebd5bca49d\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") "
Jul 28 01:43:47.999970 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:47.999910    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/186f4dca-65e7-49c8-8476-21ebd5bca49d-kube-rbac-proxy-sar-config\") pod \"186f4dca-65e7-49c8-8476-21ebd5bca49d\" (UID: \"186f4dca-65e7-49c8-8476-21ebd5bca49d\") "
Jul 28 01:43:48.000231 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.000201    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186f4dca-65e7-49c8-8476-21ebd5bca49d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "186f4dca-65e7-49c8-8476-21ebd5bca49d" (UID: "186f4dca-65e7-49c8-8476-21ebd5bca49d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:43:48.000298 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.000266    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186f4dca-65e7-49c8-8476-21ebd5bca49d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "186f4dca-65e7-49c8-8476-21ebd5bca49d" (UID: "186f4dca-65e7-49c8-8476-21ebd5bca49d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:43:48.000298 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.000282    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186f4dca-65e7-49c8-8476-21ebd5bca49d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "186f4dca-65e7-49c8-8476-21ebd5bca49d" (UID: "186f4dca-65e7-49c8-8476-21ebd5bca49d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:43:48.001956 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.001938    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186f4dca-65e7-49c8-8476-21ebd5bca49d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "186f4dca-65e7-49c8-8476-21ebd5bca49d" (UID: "186f4dca-65e7-49c8-8476-21ebd5bca49d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:43:48.002072 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.002050    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186f4dca-65e7-49c8-8476-21ebd5bca49d-kube-api-access-qktbw" (OuterVolumeSpecName: "kube-api-access-qktbw") pod "186f4dca-65e7-49c8-8476-21ebd5bca49d" (UID: "186f4dca-65e7-49c8-8476-21ebd5bca49d"). InnerVolumeSpecName "kube-api-access-qktbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:43:48.100304 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.100270    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/186f4dca-65e7-49c8-8476-21ebd5bca49d-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:43:48.100304 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.100297    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/186f4dca-65e7-49c8-8476-21ebd5bca49d-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:43:48.100304 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.100307    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qktbw\" (UniqueName: \"kubernetes.io/projected/186f4dca-65e7-49c8-8476-21ebd5bca49d-kube-api-access-qktbw\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:43:48.100304 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.100315    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/186f4dca-65e7-49c8-8476-21ebd5bca49d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:43:48.100589 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.100324    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/186f4dca-65e7-49c8-8476-21ebd5bca49d-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:43:48.164128 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.164092    2576 generic.go:356] "Generic (PLEG): container finished" podID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerID="7e6114d8523e284743dac4fb727a163fc533a9486811fc4208d94fc21e09859a" exitCode=0
Jul 28 01:43:48.164305 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.164133    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" event={"ID":"186f4dca-65e7-49c8-8476-21ebd5bca49d","Type":"ContainerDied","Data":"7e6114d8523e284743dac4fb727a163fc533a9486811fc4208d94fc21e09859a"}
Jul 28 01:43:48.164305 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.164148    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"
Jul 28 01:43:48.164305 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.164203    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b" event={"ID":"186f4dca-65e7-49c8-8476-21ebd5bca49d","Type":"ContainerDied","Data":"09fb271dc5049d7f3c030fb8976193e6304d4599356fed644b9457a6a056fc25"}
Jul 28 01:43:48.164305 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.164246    2576 scope.go:117] "RemoveContainer" containerID="7e6114d8523e284743dac4fb727a163fc533a9486811fc4208d94fc21e09859a"
Jul 28 01:43:48.174046 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.174031    2576 scope.go:117] "RemoveContainer" containerID="31fdf407e046931459f5a5c86fed88ffccd386c6824564fd067634d4363b4442"
Jul 28 01:43:48.181351 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.181336    2576 scope.go:117] "RemoveContainer" containerID="12cf1c5eede18e96250790bb5f571a9a65b95ac345d77f16bb7160829f4a38f0"
Jul 28 01:43:48.186846 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.186825    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"]
Jul 28 01:43:48.188715 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.188687    2576 scope.go:117] "RemoveContainer" containerID="764c4074e5cc55e897cf640094264af113f4c72db70757cd5389361d4928253a"
Jul 28 01:43:48.190309 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.190287    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-18035-predictor-7cb99b957d-f6z4b"]
Jul 28 01:43:48.195978 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.195963    2576 scope.go:117] "RemoveContainer" containerID="7e6114d8523e284743dac4fb727a163fc533a9486811fc4208d94fc21e09859a"
Jul 28 01:43:48.196245 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:43:48.196227    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6114d8523e284743dac4fb727a163fc533a9486811fc4208d94fc21e09859a\": container with ID starting with 7e6114d8523e284743dac4fb727a163fc533a9486811fc4208d94fc21e09859a not found: ID does not exist" containerID="7e6114d8523e284743dac4fb727a163fc533a9486811fc4208d94fc21e09859a"
Jul 28 01:43:48.196290 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.196255    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6114d8523e284743dac4fb727a163fc533a9486811fc4208d94fc21e09859a"} err="failed to get container status \"7e6114d8523e284743dac4fb727a163fc533a9486811fc4208d94fc21e09859a\": rpc error: code = NotFound desc = could not find container \"7e6114d8523e284743dac4fb727a163fc533a9486811fc4208d94fc21e09859a\": container with ID starting with 7e6114d8523e284743dac4fb727a163fc533a9486811fc4208d94fc21e09859a not found: ID does not exist"
Jul 28 01:43:48.196290 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.196272    2576 scope.go:117] "RemoveContainer" containerID="31fdf407e046931459f5a5c86fed88ffccd386c6824564fd067634d4363b4442"
Jul 28 01:43:48.196534 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:43:48.196513    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31fdf407e046931459f5a5c86fed88ffccd386c6824564fd067634d4363b4442\": container with ID starting with 31fdf407e046931459f5a5c86fed88ffccd386c6824564fd067634d4363b4442 not found: ID does not exist" containerID="31fdf407e046931459f5a5c86fed88ffccd386c6824564fd067634d4363b4442"
Jul 28 01:43:48.196613 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.196540    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31fdf407e046931459f5a5c86fed88ffccd386c6824564fd067634d4363b4442"} err="failed to get container status \"31fdf407e046931459f5a5c86fed88ffccd386c6824564fd067634d4363b4442\": rpc error: code = NotFound desc = could not find container \"31fdf407e046931459f5a5c86fed88ffccd386c6824564fd067634d4363b4442\": container with ID starting with 31fdf407e046931459f5a5c86fed88ffccd386c6824564fd067634d4363b4442 not found: ID does not exist"
Jul 28 01:43:48.196613 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.196556    2576 scope.go:117] "RemoveContainer" containerID="12cf1c5eede18e96250790bb5f571a9a65b95ac345d77f16bb7160829f4a38f0"
Jul 28 01:43:48.196844 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:43:48.196822    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12cf1c5eede18e96250790bb5f571a9a65b95ac345d77f16bb7160829f4a38f0\": container with ID starting with 12cf1c5eede18e96250790bb5f571a9a65b95ac345d77f16bb7160829f4a38f0 not found: ID does not exist" containerID="12cf1c5eede18e96250790bb5f571a9a65b95ac345d77f16bb7160829f4a38f0"
Jul 28 01:43:48.196914 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.196850    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12cf1c5eede18e96250790bb5f571a9a65b95ac345d77f16bb7160829f4a38f0"} err="failed to get container status \"12cf1c5eede18e96250790bb5f571a9a65b95ac345d77f16bb7160829f4a38f0\": rpc error: code = NotFound desc = could not find container \"12cf1c5eede18e96250790bb5f571a9a65b95ac345d77f16bb7160829f4a38f0\": container with ID starting with 12cf1c5eede18e96250790bb5f571a9a65b95ac345d77f16bb7160829f4a38f0 not found: ID does not exist"
Jul 28 01:43:48.196914 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.196864    2576 scope.go:117] "RemoveContainer" containerID="764c4074e5cc55e897cf640094264af113f4c72db70757cd5389361d4928253a"
Jul 28 01:43:48.197109 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:43:48.197096    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764c4074e5cc55e897cf640094264af113f4c72db70757cd5389361d4928253a\": container with ID starting with 764c4074e5cc55e897cf640094264af113f4c72db70757cd5389361d4928253a not found: ID does not exist" containerID="764c4074e5cc55e897cf640094264af113f4c72db70757cd5389361d4928253a"
Jul 28 01:43:48.197152 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.197112    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764c4074e5cc55e897cf640094264af113f4c72db70757cd5389361d4928253a"} err="failed to get container status \"764c4074e5cc55e897cf640094264af113f4c72db70757cd5389361d4928253a\": rpc error: code = NotFound desc = could not find container \"764c4074e5cc55e897cf640094264af113f4c72db70757cd5389361d4928253a\": container with ID starting with 764c4074e5cc55e897cf640094264af113f4c72db70757cd5389361d4928253a not found: ID does not exist"
Jul 28 01:43:48.435249 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:48.431530    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" path="/var/lib/kubelet/pods/186f4dca-65e7-49c8-8476-21ebd5bca49d/volumes"
Jul 28 01:43:49.080923 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:49.080886    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:43:59.079931 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:43:59.079885    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:44:09.080756 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:44:09.080662    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:44:19.080592 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:44:19.080556    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:44:29.080465 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:44:29.080422    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:44:33.426528 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:44:33.426482    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:44:43.426538 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:44:43.426502    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:44:53.426474 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:44:53.426427    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:45:03.427246 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:45:03.427205    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:45:13.426871 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:45:13.426826    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:45:23.427136 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:45:23.427097    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:45:33.427192 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:45:33.427096    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:45:37.426310 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:45:37.426274    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:45:47.426347 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:45:47.426309    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:45:57.426588 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:45:57.426551    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:45:57.930247 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:45:57.930212    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"]
Jul 28 01:45:57.930581 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:45:57.930551    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" containerID="cri-o://790310339e371cfb71f39605a25a58b5ebfa2a3d54505f9c9c3608ce843b514c" gracePeriod=30
Jul 28 01:45:57.930899 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:45:57.930865    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kube-rbac-proxy" containerID="cri-o://c1536f272c51f80a955a0718ff0d0ddc2b0ec3e70b695b2d7301aa40dffcf6ae" gracePeriod=30
Jul 28 01:45:57.991331 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:45:57.991293    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:45:58.612946 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:45:58.612915    2576 generic.go:356] "Generic (PLEG): container finished" podID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerID="c1536f272c51f80a955a0718ff0d0ddc2b0ec3e70b695b2d7301aa40dffcf6ae" exitCode=2
Jul 28 01:45:58.613309 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:45:58.612990    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" event={"ID":"104ebc58-a752-4fdc-bea2-30aec6ddb112","Type":"ContainerDied","Data":"c1536f272c51f80a955a0718ff0d0ddc2b0ec3e70b695b2d7301aa40dffcf6ae"}
Jul 28 01:45:59.075141 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:45:59.075088    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 28 01:46:03.104008 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.103955    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:03.245660 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.245625    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"]
Jul 28 01:46:03.245889 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.245878    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70bedd5b-7206-41fe-b0a0-f19456abe21f" containerName="kserve-container"
Jul 28 01:46:03.245932 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.245891    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="70bedd5b-7206-41fe-b0a0-f19456abe21f" containerName="kserve-container"
Jul 28 01:46:03.245932 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.245900    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent"
Jul 28 01:46:03.245932 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.245905    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent"
Jul 28 01:46:03.245932 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.245914    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="storage-initializer"
Jul 28 01:46:03.245932 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.245921    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="storage-initializer"
Jul 28 01:46:03.245932 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.245928    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kube-rbac-proxy"
Jul 28 01:46:03.245932 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.245933    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kube-rbac-proxy"
Jul 28 01:46:03.246144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.245946    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70bedd5b-7206-41fe-b0a0-f19456abe21f" containerName="kube-rbac-proxy"
Jul 28 01:46:03.246144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.245951    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="70bedd5b-7206-41fe-b0a0-f19456abe21f" containerName="kube-rbac-proxy"
Jul 28 01:46:03.246144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.245959    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container"
Jul 28 01:46:03.246144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.245964    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container"
Jul 28 01:46:03.246144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.246000    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="70bedd5b-7206-41fe-b0a0-f19456abe21f" containerName="kserve-container"
Jul 28 01:46:03.246144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.246006    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="70bedd5b-7206-41fe-b0a0-f19456abe21f" containerName="kube-rbac-proxy"
Jul 28 01:46:03.246144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.246012    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kserve-container"
Jul 28 01:46:03.246144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.246018    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="agent"
Jul 28 01:46:03.246144 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.246024    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="186f4dca-65e7-49c8-8476-21ebd5bca49d" containerName="kube-rbac-proxy"
Jul 28 01:46:03.248963 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.248937    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.251650 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.251625    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 01:46:03.251788 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.251648    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 01:46:03.263785 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.263760    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"]
Jul 28 01:46:03.266819 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.266801    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.269043 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.269025    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 01:46:03.271139 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.271120    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"]
Jul 28 01:46:03.277954 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.277930    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:03.282740 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.282717    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"]
Jul 28 01:46:03.350739 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.350699    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:03.381532 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.381448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08344546-16ab-455f-b3e1-90ae76708908-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.381532 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.381488    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50e181fa-64d8-4a2a-99fe-d772679735b8-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-vr79g\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.381532 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.381512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08344546-16ab-455f-b3e1-90ae76708908-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.381787 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.381552    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50e181fa-64d8-4a2a-99fe-d772679735b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-vr79g\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.381787 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.381621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08344546-16ab-455f-b3e1-90ae76708908-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.381787 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.381642    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08344546-16ab-455f-b3e1-90ae76708908-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.381787 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.381662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50e181fa-64d8-4a2a-99fe-d772679735b8-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-vr79g\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.381787 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.381679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjntf\" (UniqueName: \"kubernetes.io/projected/08344546-16ab-455f-b3e1-90ae76708908-kube-api-access-qjntf\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.381787 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.381737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50e181fa-64d8-4a2a-99fe-d772679735b8-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-vr79g\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.381997 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.381793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h9lb\" (UniqueName: \"kubernetes.io/projected/50e181fa-64d8-4a2a-99fe-d772679735b8-kube-api-access-5h9lb\") pod \"isvc-canary-create-predictor-cb85cf55c-vr79g\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.482680 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.482646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h9lb\" (UniqueName: \"kubernetes.io/projected/50e181fa-64d8-4a2a-99fe-d772679735b8-kube-api-access-5h9lb\") pod \"isvc-canary-create-predictor-cb85cf55c-vr79g\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.482865 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.482685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08344546-16ab-455f-b3e1-90ae76708908-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.482865 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.482710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50e181fa-64d8-4a2a-99fe-d772679735b8-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-vr79g\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.482865 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.482729    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08344546-16ab-455f-b3e1-90ae76708908-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.482865 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.482745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50e181fa-64d8-4a2a-99fe-d772679735b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-vr79g\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.482865 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.482797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08344546-16ab-455f-b3e1-90ae76708908-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.482865 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.482817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08344546-16ab-455f-b3e1-90ae76708908-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.482865 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.482834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50e181fa-64d8-4a2a-99fe-d772679735b8-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-vr79g\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.482865 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.482856    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjntf\" (UniqueName: \"kubernetes.io/projected/08344546-16ab-455f-b3e1-90ae76708908-kube-api-access-qjntf\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.483267 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.482891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50e181fa-64d8-4a2a-99fe-d772679735b8-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-vr79g\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.483267 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.483243    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08344546-16ab-455f-b3e1-90ae76708908-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.483529 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.483497    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50e181fa-64d8-4a2a-99fe-d772679735b8-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-vr79g\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.483667 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.483553    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08344546-16ab-455f-b3e1-90ae76708908-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.483667 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.483576    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50e181fa-64d8-4a2a-99fe-d772679735b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-vr79g\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.483667 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.483597    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50e181fa-64d8-4a2a-99fe-d772679735b8-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-vr79g\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.483667 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.483613    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08344546-16ab-455f-b3e1-90ae76708908-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.485577 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.485556    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50e181fa-64d8-4a2a-99fe-d772679735b8-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-vr79g\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.485693 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.485676    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08344546-16ab-455f-b3e1-90ae76708908-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.490812 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.490793    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjntf\" (UniqueName: \"kubernetes.io/projected/08344546-16ab-455f-b3e1-90ae76708908-kube-api-access-qjntf\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.490922 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.490909    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h9lb\" (UniqueName: \"kubernetes.io/projected/50e181fa-64d8-4a2a-99fe-d772679735b8-kube-api-access-5h9lb\") pod \"isvc-canary-create-predictor-cb85cf55c-vr79g\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.565176 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.565136    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:03.576272 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.576242    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:03.709872 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.709850    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"]
Jul 28 01:46:03.712278 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:46:03.712246    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e181fa_64d8_4a2a_99fe_d772679735b8.slice/crio-b1c917cfd820542fca863bcba1a16bf58b18561da847ceed345a35d4c404f95b WatchSource:0}: Error finding container b1c917cfd820542fca863bcba1a16bf58b18561da847ceed345a35d4c404f95b: Status 404 returned error can't find the container with id b1c917cfd820542fca863bcba1a16bf58b18561da847ceed345a35d4c404f95b
Jul 28 01:46:03.727371 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:03.727349    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"]
Jul 28 01:46:03.730135 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:46:03.730088    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08344546_16ab_455f_b3e1_90ae76708908.slice/crio-ee42928102a18508dbbc8a2264f1afff633c548325f54959b8cecae0d851fe03 WatchSource:0}: Error finding container ee42928102a18508dbbc8a2264f1afff633c548325f54959b8cecae0d851fe03: Status 404 returned error can't find the container with id ee42928102a18508dbbc8a2264f1afff633c548325f54959b8cecae0d851fe03
Jul 28 01:46:04.075620 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:04.075582    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 28 01:46:04.637455 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:04.637414    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" event={"ID":"08344546-16ab-455f-b3e1-90ae76708908","Type":"ContainerStarted","Data":"05a45cb51b3e7ff6d8733d9748ea096ac3ed871d3537926c793defc58d52baed"}
Jul 28 01:46:04.637874 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:04.637477    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" event={"ID":"08344546-16ab-455f-b3e1-90ae76708908","Type":"ContainerStarted","Data":"ee42928102a18508dbbc8a2264f1afff633c548325f54959b8cecae0d851fe03"}
Jul 28 01:46:04.638847 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:04.638809    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" event={"ID":"50e181fa-64d8-4a2a-99fe-d772679735b8","Type":"ContainerStarted","Data":"1f22abd5157072597d7c5a078a1bfa2ad92185f43698345439c3b9f787b228ca"}
Jul 28 01:46:04.638966 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:04.638853    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" event={"ID":"50e181fa-64d8-4a2a-99fe-d772679735b8","Type":"ContainerStarted","Data":"b1c917cfd820542fca863bcba1a16bf58b18561da847ceed345a35d4c404f95b"}
Jul 28 01:46:07.427175 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:07.427124    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:46:07.650616 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:07.650583    2576 generic.go:356] "Generic (PLEG): container finished" podID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerID="1f22abd5157072597d7c5a078a1bfa2ad92185f43698345439c3b9f787b228ca" exitCode=0
Jul 28 01:46:07.650771 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:07.650654    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" event={"ID":"50e181fa-64d8-4a2a-99fe-d772679735b8","Type":"ContainerDied","Data":"1f22abd5157072597d7c5a078a1bfa2ad92185f43698345439c3b9f787b228ca"}
Jul 28 01:46:07.652330 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:07.652310    2576 generic.go:356] "Generic (PLEG): container finished" podID="08344546-16ab-455f-b3e1-90ae76708908" containerID="05a45cb51b3e7ff6d8733d9748ea096ac3ed871d3537926c793defc58d52baed" exitCode=0
Jul 28 01:46:07.652444 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:07.652362    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" event={"ID":"08344546-16ab-455f-b3e1-90ae76708908","Type":"ContainerDied","Data":"05a45cb51b3e7ff6d8733d9748ea096ac3ed871d3537926c793defc58d52baed"}
Jul 28 01:46:07.970692 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:07.970633    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:46:08.121540 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.121500    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t57pb\" (UniqueName: \"kubernetes.io/projected/104ebc58-a752-4fdc-bea2-30aec6ddb112-kube-api-access-t57pb\") pod \"104ebc58-a752-4fdc-bea2-30aec6ddb112\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") "
Jul 28 01:46:08.121540 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.121556    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104ebc58-a752-4fdc-bea2-30aec6ddb112-proxy-tls\") pod \"104ebc58-a752-4fdc-bea2-30aec6ddb112\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") "
Jul 28 01:46:08.121788 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.121589    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104ebc58-a752-4fdc-bea2-30aec6ddb112-kube-rbac-proxy-sar-config\") pod \"104ebc58-a752-4fdc-bea2-30aec6ddb112\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") "
Jul 28 01:46:08.121788 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.121614    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/104ebc58-a752-4fdc-bea2-30aec6ddb112-cabundle-cert\") pod \"104ebc58-a752-4fdc-bea2-30aec6ddb112\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") "
Jul 28 01:46:08.121788 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.121634    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/104ebc58-a752-4fdc-bea2-30aec6ddb112-kserve-provision-location\") pod \"104ebc58-a752-4fdc-bea2-30aec6ddb112\" (UID: \"104ebc58-a752-4fdc-bea2-30aec6ddb112\") "
Jul 28 01:46:08.121977 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.121953    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/104ebc58-a752-4fdc-bea2-30aec6ddb112-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "104ebc58-a752-4fdc-bea2-30aec6ddb112" (UID: "104ebc58-a752-4fdc-bea2-30aec6ddb112"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:46:08.122038 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.121959    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104ebc58-a752-4fdc-bea2-30aec6ddb112-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "104ebc58-a752-4fdc-bea2-30aec6ddb112" (UID: "104ebc58-a752-4fdc-bea2-30aec6ddb112"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:46:08.122038 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.121954    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104ebc58-a752-4fdc-bea2-30aec6ddb112-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "104ebc58-a752-4fdc-bea2-30aec6ddb112" (UID: "104ebc58-a752-4fdc-bea2-30aec6ddb112"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:46:08.123724 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.123702    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/104ebc58-a752-4fdc-bea2-30aec6ddb112-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "104ebc58-a752-4fdc-bea2-30aec6ddb112" (UID: "104ebc58-a752-4fdc-bea2-30aec6ddb112"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:46:08.123772 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.123734    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104ebc58-a752-4fdc-bea2-30aec6ddb112-kube-api-access-t57pb" (OuterVolumeSpecName: "kube-api-access-t57pb") pod "104ebc58-a752-4fdc-bea2-30aec6ddb112" (UID: "104ebc58-a752-4fdc-bea2-30aec6ddb112"). InnerVolumeSpecName "kube-api-access-t57pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:46:08.222754 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.222722    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104ebc58-a752-4fdc-bea2-30aec6ddb112-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:46:08.222754 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.222750    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/104ebc58-a752-4fdc-bea2-30aec6ddb112-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:46:08.222914 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.222766    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/104ebc58-a752-4fdc-bea2-30aec6ddb112-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:46:08.222914 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.222783    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t57pb\" (UniqueName: \"kubernetes.io/projected/104ebc58-a752-4fdc-bea2-30aec6ddb112-kube-api-access-t57pb\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:46:08.222914 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.222797    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104ebc58-a752-4fdc-bea2-30aec6ddb112-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:46:08.657912 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.657878    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" event={"ID":"08344546-16ab-455f-b3e1-90ae76708908","Type":"ContainerStarted","Data":"2f43ee2937920d9c34af511ebe28d6c19c1532b284d81961d3b4f64cde509d12"}
Jul 28 01:46:08.658453 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.657948    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" event={"ID":"08344546-16ab-455f-b3e1-90ae76708908","Type":"ContainerStarted","Data":"54f874526ab421dc8398249fd251576c734f31611bf0fed797b61ac7352a2d7e"}
Jul 28 01:46:08.658453 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.658243    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:08.659766 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.659742    2576 generic.go:356] "Generic (PLEG): container finished" podID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerID="790310339e371cfb71f39605a25a58b5ebfa2a3d54505f9c9c3608ce843b514c" exitCode=0
Jul 28 01:46:08.659904 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.659788    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"
Jul 28 01:46:08.659904 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.659822    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" event={"ID":"104ebc58-a752-4fdc-bea2-30aec6ddb112","Type":"ContainerDied","Data":"790310339e371cfb71f39605a25a58b5ebfa2a3d54505f9c9c3608ce843b514c"}
Jul 28 01:46:08.659904 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.659882    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj" event={"ID":"104ebc58-a752-4fdc-bea2-30aec6ddb112","Type":"ContainerDied","Data":"054f0fc4311dd1e525a7bab7eda35a8a880c1ecc31b21c0970aa682ee9dd4711"}
Jul 28 01:46:08.660053 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.659911    2576 scope.go:117] "RemoveContainer" containerID="c1536f272c51f80a955a0718ff0d0ddc2b0ec3e70b695b2d7301aa40dffcf6ae"
Jul 28 01:46:08.662028 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.662005    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" event={"ID":"50e181fa-64d8-4a2a-99fe-d772679735b8","Type":"ContainerStarted","Data":"b0891d3eb7319f52ea426ff0f797d4f78c764491725304adeee864d544b6e83d"}
Jul 28 01:46:08.662142 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.662059    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" event={"ID":"50e181fa-64d8-4a2a-99fe-d772679735b8","Type":"ContainerStarted","Data":"ede5d11e1b4373b7cf0149b1af4bd8a6464a9074bda2fa69f5e28320d1559690"}
Jul 28 01:46:08.662604 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.662570    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:08.662700 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.662621    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:08.664003 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.663980    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:46:08.668240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.668211    2576 scope.go:117] "RemoveContainer" containerID="790310339e371cfb71f39605a25a58b5ebfa2a3d54505f9c9c3608ce843b514c"
Jul 28 01:46:08.675228 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.675212    2576 scope.go:117] "RemoveContainer" containerID="1fd4898fb1beb1daf63fbf7bb84c93e67f1e844c5df939c198550673350155e1"
Jul 28 01:46:08.681118 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.681064    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" podStartSLOduration=2.241990192 podStartE2EDuration="5.68104644s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.439056248s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:46:08.679422526 +0000 UTC m=+1198.850461564" watchObservedRunningTime="2026-07-28 01:46:08.68104644 +0000 UTC m=+1198.852085480"
Jul 28 01:46:08.684347 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.684327    2576 scope.go:117] "RemoveContainer" containerID="c1536f272c51f80a955a0718ff0d0ddc2b0ec3e70b695b2d7301aa40dffcf6ae"
Jul 28 01:46:08.684658 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:46:08.684636    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1536f272c51f80a955a0718ff0d0ddc2b0ec3e70b695b2d7301aa40dffcf6ae\": container with ID starting with c1536f272c51f80a955a0718ff0d0ddc2b0ec3e70b695b2d7301aa40dffcf6ae not found: ID does not exist" containerID="c1536f272c51f80a955a0718ff0d0ddc2b0ec3e70b695b2d7301aa40dffcf6ae"
Jul 28 01:46:08.684737 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.684669    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1536f272c51f80a955a0718ff0d0ddc2b0ec3e70b695b2d7301aa40dffcf6ae"} err="failed to get container status \"c1536f272c51f80a955a0718ff0d0ddc2b0ec3e70b695b2d7301aa40dffcf6ae\": rpc error: code = NotFound desc = could not find container \"c1536f272c51f80a955a0718ff0d0ddc2b0ec3e70b695b2d7301aa40dffcf6ae\": container with ID starting with c1536f272c51f80a955a0718ff0d0ddc2b0ec3e70b695b2d7301aa40dffcf6ae not found: ID does not exist"
Jul 28 01:46:08.684737 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.684688    2576 scope.go:117] "RemoveContainer" containerID="790310339e371cfb71f39605a25a58b5ebfa2a3d54505f9c9c3608ce843b514c"
Jul 28 01:46:08.684917 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:46:08.684896    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790310339e371cfb71f39605a25a58b5ebfa2a3d54505f9c9c3608ce843b514c\": container with ID starting with 790310339e371cfb71f39605a25a58b5ebfa2a3d54505f9c9c3608ce843b514c not found: ID does not exist" containerID="790310339e371cfb71f39605a25a58b5ebfa2a3d54505f9c9c3608ce843b514c"
Jul 28 01:46:08.684980 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.684924    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790310339e371cfb71f39605a25a58b5ebfa2a3d54505f9c9c3608ce843b514c"} err="failed to get container status \"790310339e371cfb71f39605a25a58b5ebfa2a3d54505f9c9c3608ce843b514c\": rpc error: code = NotFound desc = could not find container \"790310339e371cfb71f39605a25a58b5ebfa2a3d54505f9c9c3608ce843b514c\": container with ID starting with 790310339e371cfb71f39605a25a58b5ebfa2a3d54505f9c9c3608ce843b514c not found: ID does not exist"
Jul 28 01:46:08.684980 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.684946    2576 scope.go:117] "RemoveContainer" containerID="1fd4898fb1beb1daf63fbf7bb84c93e67f1e844c5df939c198550673350155e1"
Jul 28 01:46:08.685173 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:46:08.685157    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd4898fb1beb1daf63fbf7bb84c93e67f1e844c5df939c198550673350155e1\": container with ID starting with 1fd4898fb1beb1daf63fbf7bb84c93e67f1e844c5df939c198550673350155e1 not found: ID does not exist" containerID="1fd4898fb1beb1daf63fbf7bb84c93e67f1e844c5df939c198550673350155e1"
Jul 28 01:46:08.685240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.685176    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd4898fb1beb1daf63fbf7bb84c93e67f1e844c5df939c198550673350155e1"} err="failed to get container status \"1fd4898fb1beb1daf63fbf7bb84c93e67f1e844c5df939c198550673350155e1\": rpc error: code = NotFound desc = could not find container \"1fd4898fb1beb1daf63fbf7bb84c93e67f1e844c5df939c198550673350155e1\": container with ID starting with 1fd4898fb1beb1daf63fbf7bb84c93e67f1e844c5df939c198550673350155e1 not found: ID does not exist"
Jul 28 01:46:08.692080 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.692057    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"]
Jul 28 01:46:08.696510 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.696490    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6ab94-predictor-65fc4b98fd-455rj"]
Jul 28 01:46:08.712816 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:08.712772    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" podStartSLOduration=1.903836506 podStartE2EDuration="5.712757636s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.80892113s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:46:08.711702712 +0000 UTC m=+1198.882741748" watchObservedRunningTime="2026-07-28 01:46:08.712757636 +0000 UTC m=+1198.883796672"
Jul 28 01:46:09.671241 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:09.671208    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:09.671660 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:09.671273    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:46:09.672564 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:09.672541    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:46:10.432170 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:10.432136    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" path="/var/lib/kubelet/pods/104ebc58-a752-4fdc-bea2-30aec6ddb112/volumes"
Jul 28 01:46:10.675573 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:10.675477    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:46:14.676456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:14.676419    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:46:14.676986 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:14.676961    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:46:15.680779 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:15.680748    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:46:15.681444 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:15.681411    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:46:24.677547 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:24.677503    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:46:25.681499 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:25.681452    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:46:34.677166 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:34.677122    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:46:35.681589 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:35.681550    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:46:44.677865 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:44.677819    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:46:45.681806 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:45.681763    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:46:48.180674 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:48.180637    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:54.677121 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:54.677079    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:46:55.682286 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:46:55.682242    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:47:04.677443 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:04.677335    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:47:05.681947 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:05.681792    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:47:14.677556 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:14.677506    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:47:15.681428 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:15.681368    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:47:19.426707 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:19.426665    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:47:23.427558 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:23.427524    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:47:29.427587 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:29.427557    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:47:33.197721 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:33.197651    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"]
Jul 28 01:47:33.198184 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:33.198058    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kserve-container" containerID="cri-o://54f874526ab421dc8398249fd251576c734f31611bf0fed797b61ac7352a2d7e" gracePeriod=30
Jul 28 01:47:33.198338 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:33.198280    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kube-rbac-proxy" containerID="cri-o://2f43ee2937920d9c34af511ebe28d6c19c1532b284d81961d3b4f64cde509d12" gracePeriod=30
Jul 28 01:47:33.207277 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:33.207237    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"]
Jul 28 01:47:33.207795 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:33.207608    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kserve-container" containerID="cri-o://ede5d11e1b4373b7cf0149b1af4bd8a6464a9074bda2fa69f5e28320d1559690" gracePeriod=30
Jul 28 01:47:33.207795 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:33.207696    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kube-rbac-proxy" containerID="cri-o://b0891d3eb7319f52ea426ff0f797d4f78c764491725304adeee864d544b6e83d" gracePeriod=30
Jul 28 01:47:33.228016 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:33.227986    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:33.426993 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:33.426958    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:47:33.963473 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:33.963437    2576 generic.go:356] "Generic (PLEG): container finished" podID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerID="b0891d3eb7319f52ea426ff0f797d4f78c764491725304adeee864d544b6e83d" exitCode=2
Jul 28 01:47:33.963659 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:33.963514    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" event={"ID":"50e181fa-64d8-4a2a-99fe-d772679735b8","Type":"ContainerDied","Data":"b0891d3eb7319f52ea426ff0f797d4f78c764491725304adeee864d544b6e83d"}
Jul 28 01:47:33.965430 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:33.965409    2576 generic.go:356] "Generic (PLEG): container finished" podID="08344546-16ab-455f-b3e1-90ae76708908" containerID="2f43ee2937920d9c34af511ebe28d6c19c1532b284d81961d3b4f64cde509d12" exitCode=2
Jul 28 01:47:33.965529 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:33.965445    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" event={"ID":"08344546-16ab-455f-b3e1-90ae76708908","Type":"ContainerDied","Data":"2f43ee2937920d9c34af511ebe28d6c19c1532b284d81961d3b4f64cde509d12"}
Jul 28 01:47:34.672115 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:34.672078    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 28 01:47:35.676216 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:35.676161    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 28 01:47:37.988872 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:37.988837    2576 generic.go:356] "Generic (PLEG): container finished" podID="08344546-16ab-455f-b3e1-90ae76708908" containerID="54f874526ab421dc8398249fd251576c734f31611bf0fed797b61ac7352a2d7e" exitCode=0
Jul 28 01:47:37.989204 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:37.988884    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" event={"ID":"08344546-16ab-455f-b3e1-90ae76708908","Type":"ContainerDied","Data":"54f874526ab421dc8398249fd251576c734f31611bf0fed797b61ac7352a2d7e"}
Jul 28 01:47:38.046822 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.046794    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:47:38.161895 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.161869    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:47:38.202288 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.202258    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08344546-16ab-455f-b3e1-90ae76708908-kube-rbac-proxy-sar-config\") pod \"08344546-16ab-455f-b3e1-90ae76708908\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") "
Jul 28 01:47:38.202288 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.202287    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjntf\" (UniqueName: \"kubernetes.io/projected/08344546-16ab-455f-b3e1-90ae76708908-kube-api-access-qjntf\") pod \"08344546-16ab-455f-b3e1-90ae76708908\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") "
Jul 28 01:47:38.202513 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.202335    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08344546-16ab-455f-b3e1-90ae76708908-cabundle-cert\") pod \"08344546-16ab-455f-b3e1-90ae76708908\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") "
Jul 28 01:47:38.202513 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.202393    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08344546-16ab-455f-b3e1-90ae76708908-proxy-tls\") pod \"08344546-16ab-455f-b3e1-90ae76708908\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") "
Jul 28 01:47:38.202513 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.202418    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08344546-16ab-455f-b3e1-90ae76708908-kserve-provision-location\") pod \"08344546-16ab-455f-b3e1-90ae76708908\" (UID: \"08344546-16ab-455f-b3e1-90ae76708908\") "
Jul 28 01:47:38.202771 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.202744    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08344546-16ab-455f-b3e1-90ae76708908-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "08344546-16ab-455f-b3e1-90ae76708908" (UID: "08344546-16ab-455f-b3e1-90ae76708908"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:47:38.202845 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.202796    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08344546-16ab-455f-b3e1-90ae76708908-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "08344546-16ab-455f-b3e1-90ae76708908" (UID: "08344546-16ab-455f-b3e1-90ae76708908"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:47:38.202845 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.202826    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08344546-16ab-455f-b3e1-90ae76708908-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "08344546-16ab-455f-b3e1-90ae76708908" (UID: "08344546-16ab-455f-b3e1-90ae76708908"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:47:38.204566 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.204542    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08344546-16ab-455f-b3e1-90ae76708908-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "08344546-16ab-455f-b3e1-90ae76708908" (UID: "08344546-16ab-455f-b3e1-90ae76708908"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:47:38.204566 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.204550    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08344546-16ab-455f-b3e1-90ae76708908-kube-api-access-qjntf" (OuterVolumeSpecName: "kube-api-access-qjntf") pod "08344546-16ab-455f-b3e1-90ae76708908" (UID: "08344546-16ab-455f-b3e1-90ae76708908"). InnerVolumeSpecName "kube-api-access-qjntf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:47:38.303058 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.303025    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50e181fa-64d8-4a2a-99fe-d772679735b8-cabundle-cert\") pod \"50e181fa-64d8-4a2a-99fe-d772679735b8\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") "
Jul 28 01:47:38.303058 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.303072    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5h9lb\" (UniqueName: \"kubernetes.io/projected/50e181fa-64d8-4a2a-99fe-d772679735b8-kube-api-access-5h9lb\") pod \"50e181fa-64d8-4a2a-99fe-d772679735b8\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") "
Jul 28 01:47:38.303299 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.303089    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50e181fa-64d8-4a2a-99fe-d772679735b8-kube-rbac-proxy-sar-config\") pod \"50e181fa-64d8-4a2a-99fe-d772679735b8\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") "
Jul 28 01:47:38.303299 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.303128    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50e181fa-64d8-4a2a-99fe-d772679735b8-kserve-provision-location\") pod \"50e181fa-64d8-4a2a-99fe-d772679735b8\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") "
Jul 28 01:47:38.303299 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.303250    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50e181fa-64d8-4a2a-99fe-d772679735b8-proxy-tls\") pod \"50e181fa-64d8-4a2a-99fe-d772679735b8\" (UID: \"50e181fa-64d8-4a2a-99fe-d772679735b8\") "
Jul 28 01:47:38.303486 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.303460    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e181fa-64d8-4a2a-99fe-d772679735b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50e181fa-64d8-4a2a-99fe-d772679735b8" (UID: "50e181fa-64d8-4a2a-99fe-d772679735b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:47:38.303486 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.303470    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e181fa-64d8-4a2a-99fe-d772679735b8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "50e181fa-64d8-4a2a-99fe-d772679735b8" (UID: "50e181fa-64d8-4a2a-99fe-d772679735b8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:47:38.303602 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.303487    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08344546-16ab-455f-b3e1-90ae76708908-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:47:38.303602 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.303492    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e181fa-64d8-4a2a-99fe-d772679735b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50e181fa-64d8-4a2a-99fe-d772679735b8" (UID: "50e181fa-64d8-4a2a-99fe-d772679735b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:47:38.303602 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.303509    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08344546-16ab-455f-b3e1-90ae76708908-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:47:38.303602 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.303524    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08344546-16ab-455f-b3e1-90ae76708908-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:47:38.303602 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.303539    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjntf\" (UniqueName: \"kubernetes.io/projected/08344546-16ab-455f-b3e1-90ae76708908-kube-api-access-qjntf\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:47:38.303602 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.303552    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08344546-16ab-455f-b3e1-90ae76708908-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:47:38.305716 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.305691    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e181fa-64d8-4a2a-99fe-d772679735b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50e181fa-64d8-4a2a-99fe-d772679735b8" (UID: "50e181fa-64d8-4a2a-99fe-d772679735b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:47:38.305716 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.305707    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e181fa-64d8-4a2a-99fe-d772679735b8-kube-api-access-5h9lb" (OuterVolumeSpecName: "kube-api-access-5h9lb") pod "50e181fa-64d8-4a2a-99fe-d772679735b8" (UID: "50e181fa-64d8-4a2a-99fe-d772679735b8"). InnerVolumeSpecName "kube-api-access-5h9lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:47:38.310045 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.310019    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:38.369542 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369508    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"]
Jul 28 01:47:38.369748 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369737    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container"
Jul 28 01:47:38.369793 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369750    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container"
Jul 28 01:47:38.369793 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369763    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kube-rbac-proxy"
Jul 28 01:47:38.369793 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369768    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kube-rbac-proxy"
Jul 28 01:47:38.369793 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369776    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="storage-initializer"
Jul 28 01:47:38.369793 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369782    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="storage-initializer"
Jul 28 01:47:38.369793 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369788    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kserve-container"
Jul 28 01:47:38.369793 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369793    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kserve-container"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369800    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kserve-container"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369805    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kserve-container"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369813    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kube-rbac-proxy"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369818    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kube-rbac-proxy"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369823    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kube-rbac-proxy"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369828    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kube-rbac-proxy"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369834    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="storage-initializer"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369839    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="storage-initializer"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369846    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="storage-initializer"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369850    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="storage-initializer"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369887    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kube-rbac-proxy"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369894    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kserve-container"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369900    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kube-rbac-proxy"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369905    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="104ebc58-a752-4fdc-bea2-30aec6ddb112" containerName="kserve-container"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369910    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerName="kserve-container"
Jul 28 01:47:38.369987 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.369916    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="08344546-16ab-455f-b3e1-90ae76708908" containerName="kube-rbac-proxy"
Jul 28 01:47:38.373963 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.373942    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.376017 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.375999    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 01:47:38.376219 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.376206    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 01:47:38.380794 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.380773    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"]
Jul 28 01:47:38.384172 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.384136    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"]
Jul 28 01:47:38.384298 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.384276    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.385503 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.385476    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:38.386334 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.386298    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 01:47:38.402128 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.402069    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"]
Jul 28 01:47:38.403875 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.403855    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50e181fa-64d8-4a2a-99fe-d772679735b8-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:47:38.403968 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.403879    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50e181fa-64d8-4a2a-99fe-d772679735b8-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:47:38.403968 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.403894    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50e181fa-64d8-4a2a-99fe-d772679735b8-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:47:38.403968 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.403908    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h9lb\" (UniqueName: \"kubernetes.io/projected/50e181fa-64d8-4a2a-99fe-d772679735b8-kube-api-access-5h9lb\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:47:38.403968 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.403920    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50e181fa-64d8-4a2a-99fe-d772679735b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:47:38.454838 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.454803    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:38.504854 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.504825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svgtk\" (UniqueName: \"kubernetes.io/projected/a80ae38c-a41e-43de-ad68-07b8a635956c-kube-api-access-svgtk\") pod \"isvc-canary-promote-predictor-d9759ffd5-85vv5\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.505020 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.504862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a80ae38c-a41e-43de-ad68-07b8a635956c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-85vv5\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.505020 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.504887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eb29efa-bb9d-4001-9feb-120e7e6c7616-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dhwdx\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.505020 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.504939    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0eb29efa-bb9d-4001-9feb-120e7e6c7616-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dhwdx\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.505020 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.504957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dhwdx\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.505020 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.504976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmwcw\" (UniqueName: \"kubernetes.io/projected/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kube-api-access-lmwcw\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dhwdx\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.505020 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.504993    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dhwdx\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.505020 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.505007    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a80ae38c-a41e-43de-ad68-07b8a635956c-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-85vv5\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.505240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.505024    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a80ae38c-a41e-43de-ad68-07b8a635956c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-85vv5\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.505240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.505094    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a80ae38c-a41e-43de-ad68-07b8a635956c-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-85vv5\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.606139 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.606104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svgtk\" (UniqueName: \"kubernetes.io/projected/a80ae38c-a41e-43de-ad68-07b8a635956c-kube-api-access-svgtk\") pod \"isvc-canary-promote-predictor-d9759ffd5-85vv5\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.606316 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.606147    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a80ae38c-a41e-43de-ad68-07b8a635956c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-85vv5\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.606398 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.606325    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eb29efa-bb9d-4001-9feb-120e7e6c7616-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dhwdx\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.606459 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.606413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0eb29efa-bb9d-4001-9feb-120e7e6c7616-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dhwdx\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.606459 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.606442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dhwdx\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.606572 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.606459    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmwcw\" (UniqueName: \"kubernetes.io/projected/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kube-api-access-lmwcw\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dhwdx\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.606572 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.606487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dhwdx\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.606572 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.606510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a80ae38c-a41e-43de-ad68-07b8a635956c-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-85vv5\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.606572 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.606542    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a80ae38c-a41e-43de-ad68-07b8a635956c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-85vv5\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.606791 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.606586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a80ae38c-a41e-43de-ad68-07b8a635956c-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-85vv5\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.606929 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.606906    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dhwdx\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.607047 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.606992    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a80ae38c-a41e-43de-ad68-07b8a635956c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-85vv5\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.607181 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.607001    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a80ae38c-a41e-43de-ad68-07b8a635956c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-85vv5\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.607240 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.607222    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a80ae38c-a41e-43de-ad68-07b8a635956c-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-85vv5\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.607686 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.607665    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0eb29efa-bb9d-4001-9feb-120e7e6c7616-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dhwdx\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.607746 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.607677    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dhwdx\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.608831 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.608805    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eb29efa-bb9d-4001-9feb-120e7e6c7616-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dhwdx\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.609129 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.609113    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a80ae38c-a41e-43de-ad68-07b8a635956c-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-85vv5\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.615016 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.614994    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmwcw\" (UniqueName: \"kubernetes.io/projected/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kube-api-access-lmwcw\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dhwdx\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.615105 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.615015    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svgtk\" (UniqueName: \"kubernetes.io/projected/a80ae38c-a41e-43de-ad68-07b8a635956c-kube-api-access-svgtk\") pod \"isvc-canary-promote-predictor-d9759ffd5-85vv5\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.634205 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.634174    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:38.685094 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.685006    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:38.694443 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.694421    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:38.822857 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.822832    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"]
Jul 28 01:47:38.824362 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:47:38.824327    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80ae38c_a41e_43de_ad68_07b8a635956c.slice/crio-206ded286362dabdcc70798859f76c45df94236e4ffef70d655e04d66548a306 WatchSource:0}: Error finding container 206ded286362dabdcc70798859f76c45df94236e4ffef70d655e04d66548a306: Status 404 returned error can't find the container with id 206ded286362dabdcc70798859f76c45df94236e4ffef70d655e04d66548a306
Jul 28 01:47:38.843601 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.843576    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"]
Jul 28 01:47:38.857360 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:47:38.857331    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb29efa_bb9d_4001_9feb_120e7e6c7616.slice/crio-44a6f18f1deea789d433080ae08212f6920fe45da61f482b1c328ba8bf4884a0 WatchSource:0}: Error finding container 44a6f18f1deea789d433080ae08212f6920fe45da61f482b1c328ba8bf4884a0: Status 404 returned error can't find the container with id 44a6f18f1deea789d433080ae08212f6920fe45da61f482b1c328ba8bf4884a0
Jul 28 01:47:38.995395 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.995355    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" event={"ID":"a80ae38c-a41e-43de-ad68-07b8a635956c","Type":"ContainerStarted","Data":"ad0a524579e41a7208133819c1deb4e65f08c14d7c4c352fa0bf4dba80f3f1da"}
Jul 28 01:47:38.995837 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.995445    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" event={"ID":"a80ae38c-a41e-43de-ad68-07b8a635956c","Type":"ContainerStarted","Data":"206ded286362dabdcc70798859f76c45df94236e4ffef70d655e04d66548a306"}
Jul 28 01:47:38.997089 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.997055    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" event={"ID":"0eb29efa-bb9d-4001-9feb-120e7e6c7616","Type":"ContainerStarted","Data":"2d1a8da2e16f800ea8c517b52d61e1d3b1fb4622abaccd3d25d4af1e0b7f2473"}
Jul 28 01:47:38.997230 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.997120    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" event={"ID":"0eb29efa-bb9d-4001-9feb-120e7e6c7616","Type":"ContainerStarted","Data":"44a6f18f1deea789d433080ae08212f6920fe45da61f482b1c328ba8bf4884a0"}
Jul 28 01:47:38.999127 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.999100    2576 generic.go:356] "Generic (PLEG): container finished" podID="50e181fa-64d8-4a2a-99fe-d772679735b8" containerID="ede5d11e1b4373b7cf0149b1af4bd8a6464a9074bda2fa69f5e28320d1559690" exitCode=0
Jul 28 01:47:38.999316 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.999155    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"
Jul 28 01:47:38.999316 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.999181    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" event={"ID":"50e181fa-64d8-4a2a-99fe-d772679735b8","Type":"ContainerDied","Data":"ede5d11e1b4373b7cf0149b1af4bd8a6464a9074bda2fa69f5e28320d1559690"}
Jul 28 01:47:38.999316 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.999240    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g" event={"ID":"50e181fa-64d8-4a2a-99fe-d772679735b8","Type":"ContainerDied","Data":"b1c917cfd820542fca863bcba1a16bf58b18561da847ceed345a35d4c404f95b"}
Jul 28 01:47:38.999316 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:38.999282    2576 scope.go:117] "RemoveContainer" containerID="b0891d3eb7319f52ea426ff0f797d4f78c764491725304adeee864d544b6e83d"
Jul 28 01:47:39.001540 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.001479    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm" event={"ID":"08344546-16ab-455f-b3e1-90ae76708908","Type":"ContainerDied","Data":"ee42928102a18508dbbc8a2264f1afff633c548325f54959b8cecae0d851fe03"}
Jul 28 01:47:39.001540 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.001495    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"
Jul 28 01:47:39.008822 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.008802    2576 scope.go:117] "RemoveContainer" containerID="ede5d11e1b4373b7cf0149b1af4bd8a6464a9074bda2fa69f5e28320d1559690"
Jul 28 01:47:39.018313 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.018292    2576 scope.go:117] "RemoveContainer" containerID="1f22abd5157072597d7c5a078a1bfa2ad92185f43698345439c3b9f787b228ca"
Jul 28 01:47:39.026485 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.026442    2576 scope.go:117] "RemoveContainer" containerID="b0891d3eb7319f52ea426ff0f797d4f78c764491725304adeee864d544b6e83d"
Jul 28 01:47:39.026866 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:47:39.026816    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0891d3eb7319f52ea426ff0f797d4f78c764491725304adeee864d544b6e83d\": container with ID starting with b0891d3eb7319f52ea426ff0f797d4f78c764491725304adeee864d544b6e83d not found: ID does not exist" containerID="b0891d3eb7319f52ea426ff0f797d4f78c764491725304adeee864d544b6e83d"
Jul 28 01:47:39.026963 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.026853    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0891d3eb7319f52ea426ff0f797d4f78c764491725304adeee864d544b6e83d"} err="failed to get container status \"b0891d3eb7319f52ea426ff0f797d4f78c764491725304adeee864d544b6e83d\": rpc error: code = NotFound desc = could not find container \"b0891d3eb7319f52ea426ff0f797d4f78c764491725304adeee864d544b6e83d\": container with ID starting with b0891d3eb7319f52ea426ff0f797d4f78c764491725304adeee864d544b6e83d not found: ID does not exist"
Jul 28 01:47:39.026963 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.026897    2576 scope.go:117] "RemoveContainer" containerID="ede5d11e1b4373b7cf0149b1af4bd8a6464a9074bda2fa69f5e28320d1559690"
Jul 28 01:47:39.027234 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:47:39.027207    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede5d11e1b4373b7cf0149b1af4bd8a6464a9074bda2fa69f5e28320d1559690\": container with ID starting with ede5d11e1b4373b7cf0149b1af4bd8a6464a9074bda2fa69f5e28320d1559690 not found: ID does not exist" containerID="ede5d11e1b4373b7cf0149b1af4bd8a6464a9074bda2fa69f5e28320d1559690"
Jul 28 01:47:39.027332 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.027243    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede5d11e1b4373b7cf0149b1af4bd8a6464a9074bda2fa69f5e28320d1559690"} err="failed to get container status \"ede5d11e1b4373b7cf0149b1af4bd8a6464a9074bda2fa69f5e28320d1559690\": rpc error: code = NotFound desc = could not find container \"ede5d11e1b4373b7cf0149b1af4bd8a6464a9074bda2fa69f5e28320d1559690\": container with ID starting with ede5d11e1b4373b7cf0149b1af4bd8a6464a9074bda2fa69f5e28320d1559690 not found: ID does not exist"
Jul 28 01:47:39.027332 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.027270    2576 scope.go:117] "RemoveContainer" containerID="1f22abd5157072597d7c5a078a1bfa2ad92185f43698345439c3b9f787b228ca"
Jul 28 01:47:39.027575 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:47:39.027538    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f22abd5157072597d7c5a078a1bfa2ad92185f43698345439c3b9f787b228ca\": container with ID starting with 1f22abd5157072597d7c5a078a1bfa2ad92185f43698345439c3b9f787b228ca not found: ID does not exist" containerID="1f22abd5157072597d7c5a078a1bfa2ad92185f43698345439c3b9f787b228ca"
Jul 28 01:47:39.027661 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.027571    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f22abd5157072597d7c5a078a1bfa2ad92185f43698345439c3b9f787b228ca"} err="failed to get container status \"1f22abd5157072597d7c5a078a1bfa2ad92185f43698345439c3b9f787b228ca\": rpc error: code = NotFound desc = could not find container \"1f22abd5157072597d7c5a078a1bfa2ad92185f43698345439c3b9f787b228ca\": container with ID starting with 1f22abd5157072597d7c5a078a1bfa2ad92185f43698345439c3b9f787b228ca not found: ID does not exist"
Jul 28 01:47:39.027661 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.027590    2576 scope.go:117] "RemoveContainer" containerID="2f43ee2937920d9c34af511ebe28d6c19c1532b284d81961d3b4f64cde509d12"
Jul 28 01:47:39.044452 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.044426    2576 scope.go:117] "RemoveContainer" containerID="54f874526ab421dc8398249fd251576c734f31611bf0fed797b61ac7352a2d7e"
Jul 28 01:47:39.048813 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.048783    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"]
Jul 28 01:47:39.052061 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.052038    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-d5nbm"]
Jul 28 01:47:39.054286 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.054267    2576 scope.go:117] "RemoveContainer" containerID="05a45cb51b3e7ff6d8733d9748ea096ac3ed871d3537926c793defc58d52baed"
Jul 28 01:47:39.062526 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.062504    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"]
Jul 28 01:47:39.068736 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:39.068709    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vr79g"]
Jul 28 01:47:40.430918 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:40.430885    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08344546-16ab-455f-b3e1-90ae76708908" path="/var/lib/kubelet/pods/08344546-16ab-455f-b3e1-90ae76708908/volumes"
Jul 28 01:47:40.431359 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:40.431346    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50e181fa-64d8-4a2a-99fe-d772679735b8" path="/var/lib/kubelet/pods/50e181fa-64d8-4a2a-99fe-d772679735b8/volumes"
Jul 28 01:47:43.017331 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:43.017289    2576 generic.go:356] "Generic (PLEG): container finished" podID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerID="ad0a524579e41a7208133819c1deb4e65f08c14d7c4c352fa0bf4dba80f3f1da" exitCode=0
Jul 28 01:47:43.017753 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:43.017359    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" event={"ID":"a80ae38c-a41e-43de-ad68-07b8a635956c","Type":"ContainerDied","Data":"ad0a524579e41a7208133819c1deb4e65f08c14d7c4c352fa0bf4dba80f3f1da"}
Jul 28 01:47:43.019000 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:43.018979    2576 generic.go:356] "Generic (PLEG): container finished" podID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerID="2d1a8da2e16f800ea8c517b52d61e1d3b1fb4622abaccd3d25d4af1e0b7f2473" exitCode=0
Jul 28 01:47:43.019109 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:43.019033    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" event={"ID":"0eb29efa-bb9d-4001-9feb-120e7e6c7616","Type":"ContainerDied","Data":"2d1a8da2e16f800ea8c517b52d61e1d3b1fb4622abaccd3d25d4af1e0b7f2473"}
Jul 28 01:47:44.030513 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:44.030467    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" event={"ID":"a80ae38c-a41e-43de-ad68-07b8a635956c","Type":"ContainerStarted","Data":"408fd762e31e08fe8b376775b1c60dc37bbd22c40e0b6ed45ddcdd6873783ad6"}
Jul 28 01:47:44.030994 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:44.030563    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" event={"ID":"a80ae38c-a41e-43de-ad68-07b8a635956c","Type":"ContainerStarted","Data":"011069fbedc5b2dc820bb516b6b2c3835df9620b0a7bc84b4a0a34f90fbe263b"}
Jul 28 01:47:44.030994 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:44.030772    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:44.032368 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:44.032348    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" event={"ID":"0eb29efa-bb9d-4001-9feb-120e7e6c7616","Type":"ContainerStarted","Data":"902727aa3eeb23fecb121141be9815284cdbf1b2dc380c9730cced6e49a8dbc8"}
Jul 28 01:47:44.032485 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:44.032418    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" event={"ID":"0eb29efa-bb9d-4001-9feb-120e7e6c7616","Type":"ContainerStarted","Data":"acabe89c74d24d63ba600c9532540d8aaae99d231ed6242c13bb29e6ce0c68f0"}
Jul 28 01:47:44.032666 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:44.032653    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:44.032702 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:44.032689    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:44.034005 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:44.033979    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:47:44.048913 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:44.048853    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" podStartSLOduration=2.303345559 podStartE2EDuration="6.048835734s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.745490175s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:47:44.047868414 +0000 UTC m=+1294.218907451" watchObservedRunningTime="2026-07-28 01:47:44.048835734 +0000 UTC m=+1294.219874772"
Jul 28 01:47:44.065130 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:44.065083    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" podStartSLOduration=2.239681671 podStartE2EDuration="6.065066298s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.825384627s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:47:44.06410656 +0000 UTC m=+1294.235145610" watchObservedRunningTime="2026-07-28 01:47:44.065066298 +0000 UTC m=+1294.236105335"
Jul 28 01:47:45.036134 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:45.036088    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:47:45.036134 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:45.036125    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:45.037200 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:45.037176    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:47:46.039235 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:46.039198    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:47:50.040997 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:50.040968    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:47:50.041490 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:50.041465    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:47:51.043984 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:51.043960    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:47:51.044501 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:51.044476    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:47:55.370919 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:47:55.370880    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:00.042352 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:48:00.042306    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:48:01.044579 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:48:01.044538    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:48:10.041834 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:48:10.041789    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:48:11.044548 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:48:11.044509    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:48:20.041963 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:48:20.041924    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:48:21.044627 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:48:21.044588    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:48:23.266962 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:48:23.266922    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:24.367420 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:48:24.367363    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:30.041865 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:48:30.041820    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:48:31.045429 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:48:31.045371    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:48:40.041634 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:48:40.041594    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:48:41.044546 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:48:41.044511    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:48:50.041739 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:48:50.041695    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:48:51.045084 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:48:51.045042    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:49:00.042289 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:00.042253    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:49:01.045539 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:01.045510    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:49:08.454234 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:08.454201    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"]
Jul 28 01:49:08.454748 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:08.454430    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kserve-container" containerID="cri-o://011069fbedc5b2dc820bb516b6b2c3835df9620b0a7bc84b4a0a34f90fbe263b" gracePeriod=30
Jul 28 01:49:08.454748 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:08.454466    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kube-rbac-proxy" containerID="cri-o://408fd762e31e08fe8b376775b1c60dc37bbd22c40e0b6ed45ddcdd6873783ad6" gracePeriod=30
Jul 28 01:49:08.467569 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:08.467539    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:49:09.336343 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:09.336309    2576 generic.go:356] "Generic (PLEG): container finished" podID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerID="408fd762e31e08fe8b376775b1c60dc37bbd22c40e0b6ed45ddcdd6873783ad6" exitCode=2
Jul 28 01:49:09.336536 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:09.336352    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" event={"ID":"a80ae38c-a41e-43de-ad68-07b8a635956c","Type":"ContainerDied","Data":"408fd762e31e08fe8b376775b1c60dc37bbd22c40e0b6ed45ddcdd6873783ad6"}
Jul 28 01:49:11.040293 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:11.040245    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 01:49:11.044574 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:11.044544    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:49:13.199410 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.199362    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:49:13.313052 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.313019    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a80ae38c-a41e-43de-ad68-07b8a635956c-kserve-provision-location\") pod \"a80ae38c-a41e-43de-ad68-07b8a635956c\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") "
Jul 28 01:49:13.313253 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.313085    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a80ae38c-a41e-43de-ad68-07b8a635956c-cabundle-cert\") pod \"a80ae38c-a41e-43de-ad68-07b8a635956c\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") "
Jul 28 01:49:13.313253 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.313123    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a80ae38c-a41e-43de-ad68-07b8a635956c-kube-rbac-proxy-sar-config\") pod \"a80ae38c-a41e-43de-ad68-07b8a635956c\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") "
Jul 28 01:49:13.313253 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.313146    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svgtk\" (UniqueName: \"kubernetes.io/projected/a80ae38c-a41e-43de-ad68-07b8a635956c-kube-api-access-svgtk\") pod \"a80ae38c-a41e-43de-ad68-07b8a635956c\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") "
Jul 28 01:49:13.313253 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.313189    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a80ae38c-a41e-43de-ad68-07b8a635956c-proxy-tls\") pod \"a80ae38c-a41e-43de-ad68-07b8a635956c\" (UID: \"a80ae38c-a41e-43de-ad68-07b8a635956c\") "
Jul 28 01:49:13.313462 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.313420    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a80ae38c-a41e-43de-ad68-07b8a635956c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a80ae38c-a41e-43de-ad68-07b8a635956c" (UID: "a80ae38c-a41e-43de-ad68-07b8a635956c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:49:13.313529 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.313505    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80ae38c-a41e-43de-ad68-07b8a635956c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a80ae38c-a41e-43de-ad68-07b8a635956c" (UID: "a80ae38c-a41e-43de-ad68-07b8a635956c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:49:13.313567 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.313522    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80ae38c-a41e-43de-ad68-07b8a635956c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a80ae38c-a41e-43de-ad68-07b8a635956c" (UID: "a80ae38c-a41e-43de-ad68-07b8a635956c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:49:13.315454 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.315424    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80ae38c-a41e-43de-ad68-07b8a635956c-kube-api-access-svgtk" (OuterVolumeSpecName: "kube-api-access-svgtk") pod "a80ae38c-a41e-43de-ad68-07b8a635956c" (UID: "a80ae38c-a41e-43de-ad68-07b8a635956c"). InnerVolumeSpecName "kube-api-access-svgtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:49:13.315565 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.315420    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a80ae38c-a41e-43de-ad68-07b8a635956c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a80ae38c-a41e-43de-ad68-07b8a635956c" (UID: "a80ae38c-a41e-43de-ad68-07b8a635956c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:49:13.350082 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.350056    2576 generic.go:356] "Generic (PLEG): container finished" podID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerID="011069fbedc5b2dc820bb516b6b2c3835df9620b0a7bc84b4a0a34f90fbe263b" exitCode=0
Jul 28 01:49:13.350241 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.350123    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"
Jul 28 01:49:13.350241 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.350150    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" event={"ID":"a80ae38c-a41e-43de-ad68-07b8a635956c","Type":"ContainerDied","Data":"011069fbedc5b2dc820bb516b6b2c3835df9620b0a7bc84b4a0a34f90fbe263b"}
Jul 28 01:49:13.350241 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.350222    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5" event={"ID":"a80ae38c-a41e-43de-ad68-07b8a635956c","Type":"ContainerDied","Data":"206ded286362dabdcc70798859f76c45df94236e4ffef70d655e04d66548a306"}
Jul 28 01:49:13.350443 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.350259    2576 scope.go:117] "RemoveContainer" containerID="408fd762e31e08fe8b376775b1c60dc37bbd22c40e0b6ed45ddcdd6873783ad6"
Jul 28 01:49:13.360440 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.360424    2576 scope.go:117] "RemoveContainer" containerID="011069fbedc5b2dc820bb516b6b2c3835df9620b0a7bc84b4a0a34f90fbe263b"
Jul 28 01:49:13.367432 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.367377    2576 scope.go:117] "RemoveContainer" containerID="ad0a524579e41a7208133819c1deb4e65f08c14d7c4c352fa0bf4dba80f3f1da"
Jul 28 01:49:13.372131 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.372111    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"]
Jul 28 01:49:13.375017 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.375002    2576 scope.go:117] "RemoveContainer" containerID="408fd762e31e08fe8b376775b1c60dc37bbd22c40e0b6ed45ddcdd6873783ad6"
Jul 28 01:49:13.375290 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:49:13.375273    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"408fd762e31e08fe8b376775b1c60dc37bbd22c40e0b6ed45ddcdd6873783ad6\": container with ID starting with 408fd762e31e08fe8b376775b1c60dc37bbd22c40e0b6ed45ddcdd6873783ad6 not found: ID does not exist" containerID="408fd762e31e08fe8b376775b1c60dc37bbd22c40e0b6ed45ddcdd6873783ad6"
Jul 28 01:49:13.375365 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.375300    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"408fd762e31e08fe8b376775b1c60dc37bbd22c40e0b6ed45ddcdd6873783ad6"} err="failed to get container status \"408fd762e31e08fe8b376775b1c60dc37bbd22c40e0b6ed45ddcdd6873783ad6\": rpc error: code = NotFound desc = could not find container \"408fd762e31e08fe8b376775b1c60dc37bbd22c40e0b6ed45ddcdd6873783ad6\": container with ID starting with 408fd762e31e08fe8b376775b1c60dc37bbd22c40e0b6ed45ddcdd6873783ad6 not found: ID does not exist"
Jul 28 01:49:13.375365 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.375317    2576 scope.go:117] "RemoveContainer" containerID="011069fbedc5b2dc820bb516b6b2c3835df9620b0a7bc84b4a0a34f90fbe263b"
Jul 28 01:49:13.375655 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:49:13.375635    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011069fbedc5b2dc820bb516b6b2c3835df9620b0a7bc84b4a0a34f90fbe263b\": container with ID starting with 011069fbedc5b2dc820bb516b6b2c3835df9620b0a7bc84b4a0a34f90fbe263b not found: ID does not exist" containerID="011069fbedc5b2dc820bb516b6b2c3835df9620b0a7bc84b4a0a34f90fbe263b"
Jul 28 01:49:13.375746 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.375658    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-85vv5"]
Jul 28 01:49:13.375746 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.375659    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011069fbedc5b2dc820bb516b6b2c3835df9620b0a7bc84b4a0a34f90fbe263b"} err="failed to get container status \"011069fbedc5b2dc820bb516b6b2c3835df9620b0a7bc84b4a0a34f90fbe263b\": rpc error: code = NotFound desc = could not find container \"011069fbedc5b2dc820bb516b6b2c3835df9620b0a7bc84b4a0a34f90fbe263b\": container with ID starting with 011069fbedc5b2dc820bb516b6b2c3835df9620b0a7bc84b4a0a34f90fbe263b not found: ID does not exist"
Jul 28 01:49:13.375746 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.375702    2576 scope.go:117] "RemoveContainer" containerID="ad0a524579e41a7208133819c1deb4e65f08c14d7c4c352fa0bf4dba80f3f1da"
Jul 28 01:49:13.375937 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:49:13.375905    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad0a524579e41a7208133819c1deb4e65f08c14d7c4c352fa0bf4dba80f3f1da\": container with ID starting with ad0a524579e41a7208133819c1deb4e65f08c14d7c4c352fa0bf4dba80f3f1da not found: ID does not exist" containerID="ad0a524579e41a7208133819c1deb4e65f08c14d7c4c352fa0bf4dba80f3f1da"
Jul 28 01:49:13.375983 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.375943    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0a524579e41a7208133819c1deb4e65f08c14d7c4c352fa0bf4dba80f3f1da"} err="failed to get container status \"ad0a524579e41a7208133819c1deb4e65f08c14d7c4c352fa0bf4dba80f3f1da\": rpc error: code = NotFound desc = could not find container \"ad0a524579e41a7208133819c1deb4e65f08c14d7c4c352fa0bf4dba80f3f1da\": container with ID starting with ad0a524579e41a7208133819c1deb4e65f08c14d7c4c352fa0bf4dba80f3f1da not found: ID does not exist"
Jul 28 01:49:13.414464 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.414431    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a80ae38c-a41e-43de-ad68-07b8a635956c-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:49:13.414464 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.414457    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a80ae38c-a41e-43de-ad68-07b8a635956c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:49:13.414464 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.414468    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svgtk\" (UniqueName: \"kubernetes.io/projected/a80ae38c-a41e-43de-ad68-07b8a635956c-kube-api-access-svgtk\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:49:13.414660 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.414477    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a80ae38c-a41e-43de-ad68-07b8a635956c-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:49:13.414660 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:13.414485    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a80ae38c-a41e-43de-ad68-07b8a635956c-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:49:14.430196 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:49:14.430165    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" path="/var/lib/kubelet/pods/a80ae38c-a41e-43de-ad68-07b8a635956c/volumes"
Jul 28 01:50:48.684197 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:48.684162    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"]
Jul 28 01:50:48.684713 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:48.684414    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kserve-container" containerID="cri-o://acabe89c74d24d63ba600c9532540d8aaae99d231ed6242c13bb29e6ce0c68f0" gracePeriod=30
Jul 28 01:50:48.684713 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:48.684449    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kube-rbac-proxy" containerID="cri-o://902727aa3eeb23fecb121141be9815284cdbf1b2dc380c9730cced6e49a8dbc8" gracePeriod=30
Jul 28 01:50:49.601103 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:49.601070    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:49.674655 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:49.674619    2576 generic.go:356] "Generic (PLEG): container finished" podID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerID="902727aa3eeb23fecb121141be9815284cdbf1b2dc380c9730cced6e49a8dbc8" exitCode=2
Jul 28 01:50:49.674817 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:49.674686    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" event={"ID":"0eb29efa-bb9d-4001-9feb-120e7e6c7616","Type":"ContainerDied","Data":"902727aa3eeb23fecb121141be9815284cdbf1b2dc380c9730cced6e49a8dbc8"}
Jul 28 01:50:50.036426 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:50.036362    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 01:50:50.041441 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:50.041414    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:50:53.526334 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.526308    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:50:53.627317 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.627237    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kube-rbac-proxy-sar-config\") pod \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") "
Jul 28 01:50:53.627317 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.627291    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kserve-provision-location\") pod \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") "
Jul 28 01:50:53.627317 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.627316    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eb29efa-bb9d-4001-9feb-120e7e6c7616-proxy-tls\") pod \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") "
Jul 28 01:50:53.627628 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.627349    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0eb29efa-bb9d-4001-9feb-120e7e6c7616-cabundle-cert\") pod \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") "
Jul 28 01:50:53.627628 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.627406    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmwcw\" (UniqueName: \"kubernetes.io/projected/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kube-api-access-lmwcw\") pod \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\" (UID: \"0eb29efa-bb9d-4001-9feb-120e7e6c7616\") "
Jul 28 01:50:53.627735 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.627667    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0eb29efa-bb9d-4001-9feb-120e7e6c7616" (UID: "0eb29efa-bb9d-4001-9feb-120e7e6c7616"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:50:53.627735 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.627696    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0eb29efa-bb9d-4001-9feb-120e7e6c7616" (UID: "0eb29efa-bb9d-4001-9feb-120e7e6c7616"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:50:53.627735 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.627718    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb29efa-bb9d-4001-9feb-120e7e6c7616-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0eb29efa-bb9d-4001-9feb-120e7e6c7616" (UID: "0eb29efa-bb9d-4001-9feb-120e7e6c7616"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:50:53.629739 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.629714    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb29efa-bb9d-4001-9feb-120e7e6c7616-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0eb29efa-bb9d-4001-9feb-120e7e6c7616" (UID: "0eb29efa-bb9d-4001-9feb-120e7e6c7616"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:50:53.629739 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.629720    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kube-api-access-lmwcw" (OuterVolumeSpecName: "kube-api-access-lmwcw") pod "0eb29efa-bb9d-4001-9feb-120e7e6c7616" (UID: "0eb29efa-bb9d-4001-9feb-120e7e6c7616"). InnerVolumeSpecName "kube-api-access-lmwcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:50:53.690303 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.690268    2576 generic.go:356] "Generic (PLEG): container finished" podID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerID="acabe89c74d24d63ba600c9532540d8aaae99d231ed6242c13bb29e6ce0c68f0" exitCode=0
Jul 28 01:50:53.690471 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.690327    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" event={"ID":"0eb29efa-bb9d-4001-9feb-120e7e6c7616","Type":"ContainerDied","Data":"acabe89c74d24d63ba600c9532540d8aaae99d231ed6242c13bb29e6ce0c68f0"}
Jul 28 01:50:53.690471 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.690401    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx" event={"ID":"0eb29efa-bb9d-4001-9feb-120e7e6c7616","Type":"ContainerDied","Data":"44a6f18f1deea789d433080ae08212f6920fe45da61f482b1c328ba8bf4884a0"}
Jul 28 01:50:53.690471 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.690331    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"
Jul 28 01:50:53.690471 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.690441    2576 scope.go:117] "RemoveContainer" containerID="902727aa3eeb23fecb121141be9815284cdbf1b2dc380c9730cced6e49a8dbc8"
Jul 28 01:50:53.699560 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.699529    2576 scope.go:117] "RemoveContainer" containerID="acabe89c74d24d63ba600c9532540d8aaae99d231ed6242c13bb29e6ce0c68f0"
Jul 28 01:50:53.706748 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.706733    2576 scope.go:117] "RemoveContainer" containerID="2d1a8da2e16f800ea8c517b52d61e1d3b1fb4622abaccd3d25d4af1e0b7f2473"
Jul 28 01:50:53.713362 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.712918    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"]
Jul 28 01:50:53.714243 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.714222    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dhwdx"]
Jul 28 01:50:53.714907 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.714886    2576 scope.go:117] "RemoveContainer" containerID="902727aa3eeb23fecb121141be9815284cdbf1b2dc380c9730cced6e49a8dbc8"
Jul 28 01:50:53.715146 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:50:53.715130    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902727aa3eeb23fecb121141be9815284cdbf1b2dc380c9730cced6e49a8dbc8\": container with ID starting with 902727aa3eeb23fecb121141be9815284cdbf1b2dc380c9730cced6e49a8dbc8 not found: ID does not exist" containerID="902727aa3eeb23fecb121141be9815284cdbf1b2dc380c9730cced6e49a8dbc8"
Jul 28 01:50:53.715198 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.715155    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902727aa3eeb23fecb121141be9815284cdbf1b2dc380c9730cced6e49a8dbc8"} err="failed to get container status \"902727aa3eeb23fecb121141be9815284cdbf1b2dc380c9730cced6e49a8dbc8\": rpc error: code = NotFound desc = could not find container \"902727aa3eeb23fecb121141be9815284cdbf1b2dc380c9730cced6e49a8dbc8\": container with ID starting with 902727aa3eeb23fecb121141be9815284cdbf1b2dc380c9730cced6e49a8dbc8 not found: ID does not exist"
Jul 28 01:50:53.715198 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.715172    2576 scope.go:117] "RemoveContainer" containerID="acabe89c74d24d63ba600c9532540d8aaae99d231ed6242c13bb29e6ce0c68f0"
Jul 28 01:50:53.715461 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:50:53.715440    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acabe89c74d24d63ba600c9532540d8aaae99d231ed6242c13bb29e6ce0c68f0\": container with ID starting with acabe89c74d24d63ba600c9532540d8aaae99d231ed6242c13bb29e6ce0c68f0 not found: ID does not exist" containerID="acabe89c74d24d63ba600c9532540d8aaae99d231ed6242c13bb29e6ce0c68f0"
Jul 28 01:50:53.715550 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.715471    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acabe89c74d24d63ba600c9532540d8aaae99d231ed6242c13bb29e6ce0c68f0"} err="failed to get container status \"acabe89c74d24d63ba600c9532540d8aaae99d231ed6242c13bb29e6ce0c68f0\": rpc error: code = NotFound desc = could not find container \"acabe89c74d24d63ba600c9532540d8aaae99d231ed6242c13bb29e6ce0c68f0\": container with ID starting with acabe89c74d24d63ba600c9532540d8aaae99d231ed6242c13bb29e6ce0c68f0 not found: ID does not exist"
Jul 28 01:50:53.715550 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.715493    2576 scope.go:117] "RemoveContainer" containerID="2d1a8da2e16f800ea8c517b52d61e1d3b1fb4622abaccd3d25d4af1e0b7f2473"
Jul 28 01:50:53.715781 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:50:53.715752    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1a8da2e16f800ea8c517b52d61e1d3b1fb4622abaccd3d25d4af1e0b7f2473\": container with ID starting with 2d1a8da2e16f800ea8c517b52d61e1d3b1fb4622abaccd3d25d4af1e0b7f2473 not found: ID does not exist" containerID="2d1a8da2e16f800ea8c517b52d61e1d3b1fb4622abaccd3d25d4af1e0b7f2473"
Jul 28 01:50:53.715822 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.715787    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1a8da2e16f800ea8c517b52d61e1d3b1fb4622abaccd3d25d4af1e0b7f2473"} err="failed to get container status \"2d1a8da2e16f800ea8c517b52d61e1d3b1fb4622abaccd3d25d4af1e0b7f2473\": rpc error: code = NotFound desc = could not find container \"2d1a8da2e16f800ea8c517b52d61e1d3b1fb4622abaccd3d25d4af1e0b7f2473\": container with ID starting with 2d1a8da2e16f800ea8c517b52d61e1d3b1fb4622abaccd3d25d4af1e0b7f2473 not found: ID does not exist"
Jul 28 01:50:53.728114 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.728094    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmwcw\" (UniqueName: \"kubernetes.io/projected/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kube-api-access-lmwcw\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:50:53.728114 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.728115    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:50:53.728229 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.728124    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0eb29efa-bb9d-4001-9feb-120e7e6c7616-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:50:53.728229 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.728133    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eb29efa-bb9d-4001-9feb-120e7e6c7616-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:50:53.728229 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.728142    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0eb29efa-bb9d-4001-9feb-120e7e6c7616-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:50:53.773031 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.773004    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:53.839396 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.839354    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:53.844913 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.844888    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"]
Jul 28 01:50:53.845133 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845122    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="storage-initializer"
Jul 28 01:50:53.845175 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845136    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="storage-initializer"
Jul 28 01:50:53.845175 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845152    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kserve-container"
Jul 28 01:50:53.845175 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845160    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kserve-container"
Jul 28 01:50:53.845175 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845168    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="storage-initializer"
Jul 28 01:50:53.845175 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845174    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="storage-initializer"
Jul 28 01:50:53.845323 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845181    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kserve-container"
Jul 28 01:50:53.845323 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845186    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kserve-container"
Jul 28 01:50:53.845323 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845194    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kube-rbac-proxy"
Jul 28 01:50:53.845323 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845198    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kube-rbac-proxy"
Jul 28 01:50:53.845323 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845204    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kube-rbac-proxy"
Jul 28 01:50:53.845323 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845209    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kube-rbac-proxy"
Jul 28 01:50:53.845323 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845250    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kserve-container"
Jul 28 01:50:53.845323 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845257    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kube-rbac-proxy"
Jul 28 01:50:53.845323 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845266    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a80ae38c-a41e-43de-ad68-07b8a635956c" containerName="kserve-container"
Jul 28 01:50:53.845323 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.845275    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" containerName="kube-rbac-proxy"
Jul 28 01:50:53.849511 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.849491    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:53.852057 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.852021    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 28 01:50:53.852057 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.852065    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 01:50:53.852057 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.852065    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 01:50:53.852057 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.852065    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 01:50:53.852057 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.852096    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 01:50:53.852057 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.852079    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cl94s\""
Jul 28 01:50:53.852057 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.852161    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 01:50:53.858825 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.858807    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"]
Jul 28 01:50:53.928002 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:53.927922    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:54.028567 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.028532    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:54.030618 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.030588    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/683d2a54-8975-4d2b-9c7b-140bd8e7608c-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.030755 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.030632    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.030755 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.030689    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.030755 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.030737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7hlv\" (UniqueName: \"kubernetes.io/projected/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kube-api-access-m7hlv\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.030866 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.030766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/683d2a54-8975-4d2b-9c7b-140bd8e7608c-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.132006 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.131961    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7hlv\" (UniqueName: \"kubernetes.io/projected/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kube-api-access-m7hlv\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.132006 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.132012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/683d2a54-8975-4d2b-9c7b-140bd8e7608c-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.132271 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.132046    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/683d2a54-8975-4d2b-9c7b-140bd8e7608c-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.132271 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.132090    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.132271 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.132113    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.132564 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.132544    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.132807 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.132787    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/683d2a54-8975-4d2b-9c7b-140bd8e7608c-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.132851 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.132837    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.134721 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.134703    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/683d2a54-8975-4d2b-9c7b-140bd8e7608c-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.140793 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.140773    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7hlv\" (UniqueName: \"kubernetes.io/projected/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kube-api-access-m7hlv\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.160281 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.160260    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:54.287072 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.287018    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"]
Jul 28 01:50:54.289962 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:50:54.289932    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683d2a54_8975_4d2b_9c7b_140bd8e7608c.slice/crio-8ff1708c328f38f1e86bed597e2ada09ed8e881781687253e8200b4749cf0742 WatchSource:0}: Error finding container 8ff1708c328f38f1e86bed597e2ada09ed8e881781687253e8200b4749cf0742: Status 404 returned error can't find the container with id 8ff1708c328f38f1e86bed597e2ada09ed8e881781687253e8200b4749cf0742
Jul 28 01:50:54.430732 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.430699    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eb29efa-bb9d-4001-9feb-120e7e6c7616" path="/var/lib/kubelet/pods/0eb29efa-bb9d-4001-9feb-120e7e6c7616/volumes"
Jul 28 01:50:54.696535 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.696500    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" event={"ID":"683d2a54-8975-4d2b-9c7b-140bd8e7608c","Type":"ContainerStarted","Data":"2e7b2b9c2c99df2ada43f3d1637bcddd34757fdaeb60e01904e68092a9d480fb"}
Jul 28 01:50:54.696906 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:54.696580    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" event={"ID":"683d2a54-8975-4d2b-9c7b-140bd8e7608c","Type":"ContainerStarted","Data":"8ff1708c328f38f1e86bed597e2ada09ed8e881781687253e8200b4749cf0742"}
Jul 28 01:50:58.711261 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:58.711221    2576 generic.go:356] "Generic (PLEG): container finished" podID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerID="2e7b2b9c2c99df2ada43f3d1637bcddd34757fdaeb60e01904e68092a9d480fb" exitCode=0
Jul 28 01:50:58.711666 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:58.711267    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" event={"ID":"683d2a54-8975-4d2b-9c7b-140bd8e7608c","Type":"ContainerDied","Data":"2e7b2b9c2c99df2ada43f3d1637bcddd34757fdaeb60e01904e68092a9d480fb"}
Jul 28 01:50:59.717444 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:59.717410    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" event={"ID":"683d2a54-8975-4d2b-9c7b-140bd8e7608c","Type":"ContainerStarted","Data":"d21716a0b3d8f662d2728a24c963b6feaf75a99069b706cbd990a5cdbf1f6be9"}
Jul 28 01:50:59.717880 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:59.717473    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" event={"ID":"683d2a54-8975-4d2b-9c7b-140bd8e7608c","Type":"ContainerStarted","Data":"b459a6caa49c29dbb126a54843119ebb9b95ba716974e244ec78c5601ea30321"}
Jul 28 01:50:59.717880 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:59.717749    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:59.717997 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:59.717890    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:50:59.719116 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:59.719092    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 01:50:59.736519 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:50:59.736468    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" podStartSLOduration=2.938727749 podStartE2EDuration="6.736454988s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.797727239s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:50:59.735150255 +0000 UTC m=+1489.906189290" watchObservedRunningTime="2026-07-28 01:50:59.736454988 +0000 UTC m=+1489.907494025"
Jul 28 01:51:00.721399 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:51:00.721350    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 01:51:02.314920 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:51:02.314883    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:51:05.725811 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:51:05.725781    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:51:05.726280 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:51:05.726258    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 01:51:15.726911 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:51:15.726870    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 01:51:25.726787 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:51:25.726746    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 01:51:35.726607 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:51:35.726505    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 01:51:45.726899 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:51:45.726862    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 01:51:55.726220 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:51:55.726182    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 01:51:56.387446 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:51:56.387415    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:52:05.726998 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:05.726960    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 01:52:15.727036 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:15.727008    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:52:23.859111 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:23.859077    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"]
Jul 28 01:52:23.859542 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:23.859327    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kserve-container" containerID="cri-o://b459a6caa49c29dbb126a54843119ebb9b95ba716974e244ec78c5601ea30321" gracePeriod=30
Jul 28 01:52:23.859542 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:23.859427    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kube-rbac-proxy" containerID="cri-o://d21716a0b3d8f662d2728a24c963b6feaf75a99069b706cbd990a5cdbf1f6be9" gracePeriod=30
Jul 28 01:52:23.884768 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:23.884742    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:52:23.939207 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:23.939182    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"]
Jul 28 01:52:23.942374 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:23.942355    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:23.944830 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:23.944812    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 01:52:23.953355 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:23.953334    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"]
Jul 28 01:52:23.997267 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:23.997241    2576 generic.go:356] "Generic (PLEG): container finished" podID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerID="d21716a0b3d8f662d2728a24c963b6feaf75a99069b706cbd990a5cdbf1f6be9" exitCode=2
Jul 28 01:52:23.997412 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:23.997293    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" event={"ID":"683d2a54-8975-4d2b-9c7b-140bd8e7608c","Type":"ContainerDied","Data":"d21716a0b3d8f662d2728a24c963b6feaf75a99069b706cbd990a5cdbf1f6be9"}
Jul 28 01:52:24.068650 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.068615    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.068824 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.068662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.068824 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.068684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp2k5\" (UniqueName: \"kubernetes.io/projected/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kube-api-access-dp2k5\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.068824 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.068778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.068960 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.068852    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.169269 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.169183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.169269 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.169225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.169508 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.169341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.169508 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.169417    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp2k5\" (UniqueName: \"kubernetes.io/projected/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kube-api-access-dp2k5\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.169508 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.169470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.169698 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.169677    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.170009 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.169990    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.170077 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.170061    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.171997 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.171980    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.178756 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.178734    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp2k5\" (UniqueName: \"kubernetes.io/projected/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kube-api-access-dp2k5\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.251970 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.251932    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:24.380136 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:24.380084    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"]
Jul 28 01:52:24.383809 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:52:24.383784    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb86c1d_7f2c_456e_806c_dbe14b66b3a3.slice/crio-71c5bf9d865ff62a2c09cee06165b74e905ee730a53998961399a8c5bf1b63d5 WatchSource:0}: Error finding container 71c5bf9d865ff62a2c09cee06165b74e905ee730a53998961399a8c5bf1b63d5: Status 404 returned error can't find the container with id 71c5bf9d865ff62a2c09cee06165b74e905ee730a53998961399a8c5bf1b63d5
Jul 28 01:52:25.002573 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:25.002536    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" event={"ID":"deb86c1d-7f2c-456e-806c-dbe14b66b3a3","Type":"ContainerStarted","Data":"f1d03d9be7a16e95475203599210c365e0cdf5d2519715ca164d923697ec7bf9"}
Jul 28 01:52:25.003008 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:25.002593    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" event={"ID":"deb86c1d-7f2c-456e-806c-dbe14b66b3a3","Type":"ContainerStarted","Data":"71c5bf9d865ff62a2c09cee06165b74e905ee730a53998961399a8c5bf1b63d5"}
Jul 28 01:52:25.722162 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:25.722118    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 01:52:25.727065 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:25.727038    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 01:52:28.711816 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:28.711793    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:52:28.905756 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:28.905671    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kserve-provision-location\") pod \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") "
Jul 28 01:52:28.905756 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:28.905710    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7hlv\" (UniqueName: \"kubernetes.io/projected/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kube-api-access-m7hlv\") pod \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") "
Jul 28 01:52:28.905756 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:28.905738    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kube-rbac-proxy-sar-config\") pod \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") "
Jul 28 01:52:28.906038 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:28.905771    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/683d2a54-8975-4d2b-9c7b-140bd8e7608c-proxy-tls\") pod \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") "
Jul 28 01:52:28.906038 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:28.905899    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/683d2a54-8975-4d2b-9c7b-140bd8e7608c-cabundle-cert\") pod \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\" (UID: \"683d2a54-8975-4d2b-9c7b-140bd8e7608c\") "
Jul 28 01:52:28.906172 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:28.906147    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "683d2a54-8975-4d2b-9c7b-140bd8e7608c" (UID: "683d2a54-8975-4d2b-9c7b-140bd8e7608c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:52:28.906216 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:28.906186    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "683d2a54-8975-4d2b-9c7b-140bd8e7608c" (UID: "683d2a54-8975-4d2b-9c7b-140bd8e7608c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:52:28.906272 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:28.906249    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683d2a54-8975-4d2b-9c7b-140bd8e7608c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "683d2a54-8975-4d2b-9c7b-140bd8e7608c" (UID: "683d2a54-8975-4d2b-9c7b-140bd8e7608c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:52:28.907975 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:28.907956    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/683d2a54-8975-4d2b-9c7b-140bd8e7608c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "683d2a54-8975-4d2b-9c7b-140bd8e7608c" (UID: "683d2a54-8975-4d2b-9c7b-140bd8e7608c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:52:28.908158 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:28.908143    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kube-api-access-m7hlv" (OuterVolumeSpecName: "kube-api-access-m7hlv") pod "683d2a54-8975-4d2b-9c7b-140bd8e7608c" (UID: "683d2a54-8975-4d2b-9c7b-140bd8e7608c"). InnerVolumeSpecName "kube-api-access-m7hlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:52:29.007283 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.007246    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:52:29.007283 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.007271    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/683d2a54-8975-4d2b-9c7b-140bd8e7608c-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:52:29.007283 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.007283    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/683d2a54-8975-4d2b-9c7b-140bd8e7608c-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:52:29.007283 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.007290    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:52:29.007568 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.007299    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7hlv\" (UniqueName: \"kubernetes.io/projected/683d2a54-8975-4d2b-9c7b-140bd8e7608c-kube-api-access-m7hlv\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:52:29.016715 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.016684    2576 generic.go:356] "Generic (PLEG): container finished" podID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerID="b459a6caa49c29dbb126a54843119ebb9b95ba716974e244ec78c5601ea30321" exitCode=0
Jul 28 01:52:29.016860 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.016737    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"
Jul 28 01:52:29.016860 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.016775    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" event={"ID":"683d2a54-8975-4d2b-9c7b-140bd8e7608c","Type":"ContainerDied","Data":"b459a6caa49c29dbb126a54843119ebb9b95ba716974e244ec78c5601ea30321"}
Jul 28 01:52:29.016860 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.016843    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv" event={"ID":"683d2a54-8975-4d2b-9c7b-140bd8e7608c","Type":"ContainerDied","Data":"8ff1708c328f38f1e86bed597e2ada09ed8e881781687253e8200b4749cf0742"}
Jul 28 01:52:29.016994 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.016872    2576 scope.go:117] "RemoveContainer" containerID="d21716a0b3d8f662d2728a24c963b6feaf75a99069b706cbd990a5cdbf1f6be9"
Jul 28 01:52:29.018263 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.018214    2576 generic.go:356] "Generic (PLEG): container finished" podID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerID="f1d03d9be7a16e95475203599210c365e0cdf5d2519715ca164d923697ec7bf9" exitCode=0
Jul 28 01:52:29.018263 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.018251    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" event={"ID":"deb86c1d-7f2c-456e-806c-dbe14b66b3a3","Type":"ContainerDied","Data":"f1d03d9be7a16e95475203599210c365e0cdf5d2519715ca164d923697ec7bf9"}
Jul 28 01:52:29.025785 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.025764    2576 scope.go:117] "RemoveContainer" containerID="b459a6caa49c29dbb126a54843119ebb9b95ba716974e244ec78c5601ea30321"
Jul 28 01:52:29.033263 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.033247    2576 scope.go:117] "RemoveContainer" containerID="2e7b2b9c2c99df2ada43f3d1637bcddd34757fdaeb60e01904e68092a9d480fb"
Jul 28 01:52:29.042269 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.042251    2576 scope.go:117] "RemoveContainer" containerID="d21716a0b3d8f662d2728a24c963b6feaf75a99069b706cbd990a5cdbf1f6be9"
Jul 28 01:52:29.042616 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:52:29.042596    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21716a0b3d8f662d2728a24c963b6feaf75a99069b706cbd990a5cdbf1f6be9\": container with ID starting with d21716a0b3d8f662d2728a24c963b6feaf75a99069b706cbd990a5cdbf1f6be9 not found: ID does not exist" containerID="d21716a0b3d8f662d2728a24c963b6feaf75a99069b706cbd990a5cdbf1f6be9"
Jul 28 01:52:29.042710 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.042625    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21716a0b3d8f662d2728a24c963b6feaf75a99069b706cbd990a5cdbf1f6be9"} err="failed to get container status \"d21716a0b3d8f662d2728a24c963b6feaf75a99069b706cbd990a5cdbf1f6be9\": rpc error: code = NotFound desc = could not find container \"d21716a0b3d8f662d2728a24c963b6feaf75a99069b706cbd990a5cdbf1f6be9\": container with ID starting with d21716a0b3d8f662d2728a24c963b6feaf75a99069b706cbd990a5cdbf1f6be9 not found: ID does not exist"
Jul 28 01:52:29.042710 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.042647    2576 scope.go:117] "RemoveContainer" containerID="b459a6caa49c29dbb126a54843119ebb9b95ba716974e244ec78c5601ea30321"
Jul 28 01:52:29.042888 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:52:29.042871    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b459a6caa49c29dbb126a54843119ebb9b95ba716974e244ec78c5601ea30321\": container with ID starting with b459a6caa49c29dbb126a54843119ebb9b95ba716974e244ec78c5601ea30321 not found: ID does not exist" containerID="b459a6caa49c29dbb126a54843119ebb9b95ba716974e244ec78c5601ea30321"
Jul 28 01:52:29.042950 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.042897    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b459a6caa49c29dbb126a54843119ebb9b95ba716974e244ec78c5601ea30321"} err="failed to get container status \"b459a6caa49c29dbb126a54843119ebb9b95ba716974e244ec78c5601ea30321\": rpc error: code = NotFound desc = could not find container \"b459a6caa49c29dbb126a54843119ebb9b95ba716974e244ec78c5601ea30321\": container with ID starting with b459a6caa49c29dbb126a54843119ebb9b95ba716974e244ec78c5601ea30321 not found: ID does not exist"
Jul 28 01:52:29.042950 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.042921    2576 scope.go:117] "RemoveContainer" containerID="2e7b2b9c2c99df2ada43f3d1637bcddd34757fdaeb60e01904e68092a9d480fb"
Jul 28 01:52:29.043131 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:52:29.043114    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7b2b9c2c99df2ada43f3d1637bcddd34757fdaeb60e01904e68092a9d480fb\": container with ID starting with 2e7b2b9c2c99df2ada43f3d1637bcddd34757fdaeb60e01904e68092a9d480fb not found: ID does not exist" containerID="2e7b2b9c2c99df2ada43f3d1637bcddd34757fdaeb60e01904e68092a9d480fb"
Jul 28 01:52:29.043176 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.043138    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7b2b9c2c99df2ada43f3d1637bcddd34757fdaeb60e01904e68092a9d480fb"} err="failed to get container status \"2e7b2b9c2c99df2ada43f3d1637bcddd34757fdaeb60e01904e68092a9d480fb\": rpc error: code = NotFound desc = could not find container \"2e7b2b9c2c99df2ada43f3d1637bcddd34757fdaeb60e01904e68092a9d480fb\": container with ID starting with 2e7b2b9c2c99df2ada43f3d1637bcddd34757fdaeb60e01904e68092a9d480fb not found: ID does not exist"
Jul 28 01:52:29.051727 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.051705    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"]
Jul 28 01:52:29.056182 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:29.056160    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-sm5cv"]
Jul 28 01:52:30.024837 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:30.024804    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" event={"ID":"deb86c1d-7f2c-456e-806c-dbe14b66b3a3","Type":"ContainerStarted","Data":"eb40d0984c2c5728c5bf79be3e4e32724df9e8c1f7b29920655ca00e78846826"}
Jul 28 01:52:30.025241 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:30.024855    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" event={"ID":"deb86c1d-7f2c-456e-806c-dbe14b66b3a3","Type":"ContainerStarted","Data":"543fb075bb2c7c5939e178ddbfd326d49690e15bec85e895ab55d7761cef4024"}
Jul 28 01:52:30.025241 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:30.025124    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:30.025315 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:30.025260    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:30.026576 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:30.026548    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:52:30.048902 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:30.048852    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" podStartSLOduration=3.221095714 podStartE2EDuration="7.04883432s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.827738606s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:52:30.048704863 +0000 UTC m=+1580.219743902" watchObservedRunningTime="2026-07-28 01:52:30.04883432 +0000 UTC m=+1580.219873359"
Jul 28 01:52:30.430808 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:30.430731    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" path="/var/lib/kubelet/pods/683d2a54-8975-4d2b-9c7b-140bd8e7608c/volumes"
Jul 28 01:52:31.027877 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:31.027828    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:52:36.032836 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:36.032807    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:52:36.033328 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:36.033301    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:52:46.034009 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:46.033962    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:52:56.033986 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:56.033948    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:52:59.074662 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:52:59.074616    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:00.620583 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:00.620493    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:06.033432 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:06.033366    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:53:16.033672 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:16.033626    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:53:26.033936 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:26.033843    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:53:36.033444 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:36.033378    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:53:43.427036 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:43.426995    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:53:43.986890 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:43.986858    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"]
Jul 28 01:53:44.023725 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:44.023693    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:44.324434 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:53:44.324331    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 01:53:44.324434 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:53:44.324424    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-proxy-tls podName:deb86c1d-7f2c-456e-806c-dbe14b66b3a3 nodeName:}" failed. No retries permitted until 2026-07-28 01:53:44.824401857 +0000 UTC m=+1654.995440871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-proxy-tls") pod "isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" (UID: "deb86c1d-7f2c-456e-806c-dbe14b66b3a3") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 01:53:44.827848 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:53:44.827813    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 01:53:44.828233 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:53:44.827887    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-proxy-tls podName:deb86c1d-7f2c-456e-806c-dbe14b66b3a3 nodeName:}" failed. No retries permitted until 2026-07-28 01:53:45.827874385 +0000 UTC m=+1655.998913400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-proxy-tls") pod "isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" (UID: "deb86c1d-7f2c-456e-806c-dbe14b66b3a3") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 01:53:45.290227 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:45.290181    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kserve-container" containerID="cri-o://543fb075bb2c7c5939e178ddbfd326d49690e15bec85e895ab55d7761cef4024" gracePeriod=30
Jul 28 01:53:45.290450 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:45.290202    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kube-rbac-proxy" containerID="cri-o://eb40d0984c2c5728c5bf79be3e4e32724df9e8c1f7b29920655ca00e78846826" gracePeriod=30
Jul 28 01:53:45.834258 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:53:45.834216    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 01:53:45.834767 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:53:45.834308    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-proxy-tls podName:deb86c1d-7f2c-456e-806c-dbe14b66b3a3 nodeName:}" failed. No retries permitted until 2026-07-28 01:53:47.834290602 +0000 UTC m=+1658.005329617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-proxy-tls") pod "isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" (UID: "deb86c1d-7f2c-456e-806c-dbe14b66b3a3") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 01:53:46.028507 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:46.028464    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 28 01:53:46.295105 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:46.295075    2576 generic.go:356] "Generic (PLEG): container finished" podID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerID="eb40d0984c2c5728c5bf79be3e4e32724df9e8c1f7b29920655ca00e78846826" exitCode=2
Jul 28 01:53:46.295281 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:46.295123    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" event={"ID":"deb86c1d-7f2c-456e-806c-dbe14b66b3a3","Type":"ContainerDied","Data":"eb40d0984c2c5728c5bf79be3e4e32724df9e8c1f7b29920655ca00e78846826"}
Jul 28 01:53:47.850261 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:53:47.850223    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 01:53:47.850761 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:53:47.850301    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-proxy-tls podName:deb86c1d-7f2c-456e-806c-dbe14b66b3a3 nodeName:}" failed. No retries permitted until 2026-07-28 01:53:51.850283402 +0000 UTC m=+1662.021322421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-proxy-tls") pod "isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" (UID: "deb86c1d-7f2c-456e-806c-dbe14b66b3a3") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 01:53:49.127968 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.127932    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:49.202230 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.202198    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:49.206880 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.206857    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"]
Jul 28 01:53:49.207104 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.207093    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="storage-initializer"
Jul 28 01:53:49.207147 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.207106    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="storage-initializer"
Jul 28 01:53:49.207147 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.207121    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kserve-container"
Jul 28 01:53:49.207147 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.207127    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kserve-container"
Jul 28 01:53:49.207147 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.207141    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kube-rbac-proxy"
Jul 28 01:53:49.207147 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.207146    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kube-rbac-proxy"
Jul 28 01:53:49.207299 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.207179    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kserve-container"
Jul 28 01:53:49.207299 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.207188    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="683d2a54-8975-4d2b-9c7b-140bd8e7608c" containerName="kube-rbac-proxy"
Jul 28 01:53:49.210074 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.210057    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.212364 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.212340    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 01:53:49.212583 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.212570    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 01:53:49.222196 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.222175    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"]
Jul 28 01:53:49.263190 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.263152    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19af2665-2e6e-4110-9146-fa8ac990d0af-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.263190 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.263190    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19af2665-2e6e-4110-9146-fa8ac990d0af-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.263411 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.263213    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19af2665-2e6e-4110-9146-fa8ac990d0af-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.263411 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.263268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19af2665-2e6e-4110-9146-fa8ac990d0af-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.263411 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.263304    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwdhx\" (UniqueName: \"kubernetes.io/projected/19af2665-2e6e-4110-9146-fa8ac990d0af-kube-api-access-pwdhx\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.364572 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.364523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19af2665-2e6e-4110-9146-fa8ac990d0af-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.364572 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.364576    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19af2665-2e6e-4110-9146-fa8ac990d0af-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.364785 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.364700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19af2665-2e6e-4110-9146-fa8ac990d0af-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.364785 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.364771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19af2665-2e6e-4110-9146-fa8ac990d0af-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.364879 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.364820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwdhx\" (UniqueName: \"kubernetes.io/projected/19af2665-2e6e-4110-9146-fa8ac990d0af-kube-api-access-pwdhx\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.364916 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:53:49.364875    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 28 01:53:49.364958 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:53:49.364940    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19af2665-2e6e-4110-9146-fa8ac990d0af-proxy-tls podName:19af2665-2e6e-4110-9146-fa8ac990d0af nodeName:}" failed. No retries permitted until 2026-07-28 01:53:49.864919172 +0000 UTC m=+1660.035958186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/19af2665-2e6e-4110-9146-fa8ac990d0af-proxy-tls") pod "isvc-canary-stop-v2-predictor-77f778d864-xkv9j" (UID: "19af2665-2e6e-4110-9146-fa8ac990d0af") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 28 01:53:49.365137 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.365117    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19af2665-2e6e-4110-9146-fa8ac990d0af-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.365279 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.365263    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19af2665-2e6e-4110-9146-fa8ac990d0af-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.365320 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.365293    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19af2665-2e6e-4110-9146-fa8ac990d0af-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.375403 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.375357    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwdhx\" (UniqueName: \"kubernetes.io/projected/19af2665-2e6e-4110-9146-fa8ac990d0af-kube-api-access-pwdhx\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.501511 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.501477    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:49.868600 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.868569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19af2665-2e6e-4110-9146-fa8ac990d0af-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.871037 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.871017    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19af2665-2e6e-4110-9146-fa8ac990d0af-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-xkv9j\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:49.936286 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:49.936255    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:53:50.070270 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.070186    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-proxy-tls\") pod \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") "
Jul 28 01:53:50.070270 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.070241    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kserve-provision-location\") pod \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") "
Jul 28 01:53:50.070270 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.070265    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-cabundle-cert\") pod \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") "
Jul 28 01:53:50.070661 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.070323    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp2k5\" (UniqueName: \"kubernetes.io/projected/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kube-api-access-dp2k5\") pod \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") "
Jul 28 01:53:50.070661 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.070344    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kube-rbac-proxy-sar-config\") pod \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\" (UID: \"deb86c1d-7f2c-456e-806c-dbe14b66b3a3\") "
Jul 28 01:53:50.070661 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.070590    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "deb86c1d-7f2c-456e-806c-dbe14b66b3a3" (UID: "deb86c1d-7f2c-456e-806c-dbe14b66b3a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:53:50.070807 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.070726    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "deb86c1d-7f2c-456e-806c-dbe14b66b3a3" (UID: "deb86c1d-7f2c-456e-806c-dbe14b66b3a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:53:50.070807 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.070740    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "deb86c1d-7f2c-456e-806c-dbe14b66b3a3" (UID: "deb86c1d-7f2c-456e-806c-dbe14b66b3a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:53:50.072579 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.072555    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "deb86c1d-7f2c-456e-806c-dbe14b66b3a3" (UID: "deb86c1d-7f2c-456e-806c-dbe14b66b3a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:53:50.072579 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.072563    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kube-api-access-dp2k5" (OuterVolumeSpecName: "kube-api-access-dp2k5") pod "deb86c1d-7f2c-456e-806c-dbe14b66b3a3" (UID: "deb86c1d-7f2c-456e-806c-dbe14b66b3a3"). InnerVolumeSpecName "kube-api-access-dp2k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:53:50.120008 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.119972    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:50.171771 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.171743    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp2k5\" (UniqueName: \"kubernetes.io/projected/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kube-api-access-dp2k5\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:53:50.171771 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.171773    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:53:50.172123 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.171789    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:53:50.172123 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.171803    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:53:50.172123 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.171815    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/deb86c1d-7f2c-456e-806c-dbe14b66b3a3-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:53:50.247775 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.247744    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"]
Jul 28 01:53:50.250887 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:53:50.250861    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19af2665_2e6e_4110_9146_fa8ac990d0af.slice/crio-694bbeab602ebe5442e7922d150ad7e938d868a258982b1bf80eb49fbd75f40e WatchSource:0}: Error finding container 694bbeab602ebe5442e7922d150ad7e938d868a258982b1bf80eb49fbd75f40e: Status 404 returned error can't find the container with id 694bbeab602ebe5442e7922d150ad7e938d868a258982b1bf80eb49fbd75f40e
Jul 28 01:53:50.310517 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.310491    2576 generic.go:356] "Generic (PLEG): container finished" podID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerID="543fb075bb2c7c5939e178ddbfd326d49690e15bec85e895ab55d7761cef4024" exitCode=0
Jul 28 01:53:50.310622 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.310547    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"
Jul 28 01:53:50.310622 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.310554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" event={"ID":"deb86c1d-7f2c-456e-806c-dbe14b66b3a3","Type":"ContainerDied","Data":"543fb075bb2c7c5939e178ddbfd326d49690e15bec85e895ab55d7761cef4024"}
Jul 28 01:53:50.310739 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.310633    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt" event={"ID":"deb86c1d-7f2c-456e-806c-dbe14b66b3a3","Type":"ContainerDied","Data":"71c5bf9d865ff62a2c09cee06165b74e905ee730a53998961399a8c5bf1b63d5"}
Jul 28 01:53:50.310739 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.310675    2576 scope.go:117] "RemoveContainer" containerID="eb40d0984c2c5728c5bf79be3e4e32724df9e8c1f7b29920655ca00e78846826"
Jul 28 01:53:50.312274 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.312243    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" event={"ID":"19af2665-2e6e-4110-9146-fa8ac990d0af","Type":"ContainerStarted","Data":"220e0dc6b5c9b864bfd1d659510f8159eddedb125226cdfb1c319738b9d7078e"}
Jul 28 01:53:50.312370 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.312303    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" event={"ID":"19af2665-2e6e-4110-9146-fa8ac990d0af","Type":"ContainerStarted","Data":"694bbeab602ebe5442e7922d150ad7e938d868a258982b1bf80eb49fbd75f40e"}
Jul 28 01:53:50.318890 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.318868    2576 scope.go:117] "RemoveContainer" containerID="543fb075bb2c7c5939e178ddbfd326d49690e15bec85e895ab55d7761cef4024"
Jul 28 01:53:50.326174 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.326155    2576 scope.go:117] "RemoveContainer" containerID="f1d03d9be7a16e95475203599210c365e0cdf5d2519715ca164d923697ec7bf9"
Jul 28 01:53:50.333822 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.333806    2576 scope.go:117] "RemoveContainer" containerID="eb40d0984c2c5728c5bf79be3e4e32724df9e8c1f7b29920655ca00e78846826"
Jul 28 01:53:50.334105 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:53:50.334082    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb40d0984c2c5728c5bf79be3e4e32724df9e8c1f7b29920655ca00e78846826\": container with ID starting with eb40d0984c2c5728c5bf79be3e4e32724df9e8c1f7b29920655ca00e78846826 not found: ID does not exist" containerID="eb40d0984c2c5728c5bf79be3e4e32724df9e8c1f7b29920655ca00e78846826"
Jul 28 01:53:50.334191 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.334123    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb40d0984c2c5728c5bf79be3e4e32724df9e8c1f7b29920655ca00e78846826"} err="failed to get container status \"eb40d0984c2c5728c5bf79be3e4e32724df9e8c1f7b29920655ca00e78846826\": rpc error: code = NotFound desc = could not find container \"eb40d0984c2c5728c5bf79be3e4e32724df9e8c1f7b29920655ca00e78846826\": container with ID starting with eb40d0984c2c5728c5bf79be3e4e32724df9e8c1f7b29920655ca00e78846826 not found: ID does not exist"
Jul 28 01:53:50.334191 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.334141    2576 scope.go:117] "RemoveContainer" containerID="543fb075bb2c7c5939e178ddbfd326d49690e15bec85e895ab55d7761cef4024"
Jul 28 01:53:50.334467 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:53:50.334449    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543fb075bb2c7c5939e178ddbfd326d49690e15bec85e895ab55d7761cef4024\": container with ID starting with 543fb075bb2c7c5939e178ddbfd326d49690e15bec85e895ab55d7761cef4024 not found: ID does not exist" containerID="543fb075bb2c7c5939e178ddbfd326d49690e15bec85e895ab55d7761cef4024"
Jul 28 01:53:50.334541 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.334471    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543fb075bb2c7c5939e178ddbfd326d49690e15bec85e895ab55d7761cef4024"} err="failed to get container status \"543fb075bb2c7c5939e178ddbfd326d49690e15bec85e895ab55d7761cef4024\": rpc error: code = NotFound desc = could not find container \"543fb075bb2c7c5939e178ddbfd326d49690e15bec85e895ab55d7761cef4024\": container with ID starting with 543fb075bb2c7c5939e178ddbfd326d49690e15bec85e895ab55d7761cef4024 not found: ID does not exist"
Jul 28 01:53:50.334541 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.334491    2576 scope.go:117] "RemoveContainer" containerID="f1d03d9be7a16e95475203599210c365e0cdf5d2519715ca164d923697ec7bf9"
Jul 28 01:53:50.334777 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:53:50.334753    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d03d9be7a16e95475203599210c365e0cdf5d2519715ca164d923697ec7bf9\": container with ID starting with f1d03d9be7a16e95475203599210c365e0cdf5d2519715ca164d923697ec7bf9 not found: ID does not exist" containerID="f1d03d9be7a16e95475203599210c365e0cdf5d2519715ca164d923697ec7bf9"
Jul 28 01:53:50.334843 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.334787    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d03d9be7a16e95475203599210c365e0cdf5d2519715ca164d923697ec7bf9"} err="failed to get container status \"f1d03d9be7a16e95475203599210c365e0cdf5d2519715ca164d923697ec7bf9\": rpc error: code = NotFound desc = could not find container \"f1d03d9be7a16e95475203599210c365e0cdf5d2519715ca164d923697ec7bf9\": container with ID starting with f1d03d9be7a16e95475203599210c365e0cdf5d2519715ca164d923697ec7bf9 not found: ID does not exist"
Jul 28 01:53:50.344422 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.344373    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"]
Jul 28 01:53:50.348928 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.348903    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-v5rlt"]
Jul 28 01:53:50.430912 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:50.430883    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" path="/var/lib/kubelet/pods/deb86c1d-7f2c-456e-806c-dbe14b66b3a3/volumes"
Jul 28 01:53:54.327781 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:54.327748    2576 generic.go:356] "Generic (PLEG): container finished" podID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerID="220e0dc6b5c9b864bfd1d659510f8159eddedb125226cdfb1c319738b9d7078e" exitCode=0
Jul 28 01:53:54.328192 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:54.327812    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" event={"ID":"19af2665-2e6e-4110-9146-fa8ac990d0af","Type":"ContainerDied","Data":"220e0dc6b5c9b864bfd1d659510f8159eddedb125226cdfb1c319738b9d7078e"}
Jul 28 01:53:55.332608 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:55.332577    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" event={"ID":"19af2665-2e6e-4110-9146-fa8ac990d0af","Type":"ContainerStarted","Data":"e65c73d690174d2e1d3801cd499694ee3a412b7bed65528b97562c24eb55b3bf"}
Jul 28 01:53:55.333073 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:55.332634    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" event={"ID":"19af2665-2e6e-4110-9146-fa8ac990d0af","Type":"ContainerStarted","Data":"aa00fff6200197e9c1d2efa72ac7bf4966218335ea107a6bcca565dcdb991296"}
Jul 28 01:53:55.333073 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:55.332891    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:55.333073 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:55.333021    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:53:55.334238 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:55.334213    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:53:55.353071 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:55.352993    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" podStartSLOduration=2.734725816 podStartE2EDuration="6.352978676s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.61825286s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:53:55.352417224 +0000 UTC m=+1665.523456262" watchObservedRunningTime="2026-07-28 01:53:55.352978676 +0000 UTC m=+1665.524018115"
Jul 28 01:53:56.336578 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:53:56.336541    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:54:01.341145 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:54:01.341116    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:54:01.341599 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:54:01.341573    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:54:11.341768 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:54:11.341731    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:54:21.341640 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:54:21.341596    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:54:31.342567 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:54:31.342488    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:54:41.341687 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:54:41.341643    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:54:51.341982 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:54:51.341944    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:55:01.341957 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:01.341916    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:55:08.430582 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:08.430554    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:55:19.225055 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:19.225018    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:19.249772 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:19.249738    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"]
Jul 28 01:55:19.250015 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:19.249965    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kserve-container" containerID="cri-o://aa00fff6200197e9c1d2efa72ac7bf4966218335ea107a6bcca565dcdb991296" gracePeriod=30
Jul 28 01:55:19.250143 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:19.250055    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kube-rbac-proxy" containerID="cri-o://e65c73d690174d2e1d3801cd499694ee3a412b7bed65528b97562c24eb55b3bf" gracePeriod=30
Jul 28 01:55:19.323726 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:19.323693    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:19.608177 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:19.608141    2576 generic.go:356] "Generic (PLEG): container finished" podID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerID="e65c73d690174d2e1d3801cd499694ee3a412b7bed65528b97562c24eb55b3bf" exitCode=2
Jul 28 01:55:19.608351 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:19.608212    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" event={"ID":"19af2665-2e6e-4110-9146-fa8ac990d0af","Type":"ContainerDied","Data":"e65c73d690174d2e1d3801cd499694ee3a412b7bed65528b97562c24eb55b3bf"}
Jul 28 01:55:21.337638 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:21.337589    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 01:55:24.099482 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.099459    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:55:24.239740 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.239706    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19af2665-2e6e-4110-9146-fa8ac990d0af-cabundle-cert\") pod \"19af2665-2e6e-4110-9146-fa8ac990d0af\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") "
Jul 28 01:55:24.239921 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.239760    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19af2665-2e6e-4110-9146-fa8ac990d0af-kube-rbac-proxy-sar-config\") pod \"19af2665-2e6e-4110-9146-fa8ac990d0af\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") "
Jul 28 01:55:24.239921 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.239797    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19af2665-2e6e-4110-9146-fa8ac990d0af-kserve-provision-location\") pod \"19af2665-2e6e-4110-9146-fa8ac990d0af\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") "
Jul 28 01:55:24.239921 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.239849    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwdhx\" (UniqueName: \"kubernetes.io/projected/19af2665-2e6e-4110-9146-fa8ac990d0af-kube-api-access-pwdhx\") pod \"19af2665-2e6e-4110-9146-fa8ac990d0af\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") "
Jul 28 01:55:24.239921 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.239881    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19af2665-2e6e-4110-9146-fa8ac990d0af-proxy-tls\") pod \"19af2665-2e6e-4110-9146-fa8ac990d0af\" (UID: \"19af2665-2e6e-4110-9146-fa8ac990d0af\") "
Jul 28 01:55:24.240164 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.240129    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19af2665-2e6e-4110-9146-fa8ac990d0af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19af2665-2e6e-4110-9146-fa8ac990d0af" (UID: "19af2665-2e6e-4110-9146-fa8ac990d0af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:55:24.240217 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.240135    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19af2665-2e6e-4110-9146-fa8ac990d0af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "19af2665-2e6e-4110-9146-fa8ac990d0af" (UID: "19af2665-2e6e-4110-9146-fa8ac990d0af"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:55:24.240217 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.240141    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19af2665-2e6e-4110-9146-fa8ac990d0af-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "19af2665-2e6e-4110-9146-fa8ac990d0af" (UID: "19af2665-2e6e-4110-9146-fa8ac990d0af"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:55:24.242118 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.242092    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19af2665-2e6e-4110-9146-fa8ac990d0af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19af2665-2e6e-4110-9146-fa8ac990d0af" (UID: "19af2665-2e6e-4110-9146-fa8ac990d0af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:55:24.242451 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.242432    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19af2665-2e6e-4110-9146-fa8ac990d0af-kube-api-access-pwdhx" (OuterVolumeSpecName: "kube-api-access-pwdhx") pod "19af2665-2e6e-4110-9146-fa8ac990d0af" (UID: "19af2665-2e6e-4110-9146-fa8ac990d0af"). InnerVolumeSpecName "kube-api-access-pwdhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:55:24.340686 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.340637    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19af2665-2e6e-4110-9146-fa8ac990d0af-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:55:24.340686 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.340681    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19af2665-2e6e-4110-9146-fa8ac990d0af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:55:24.340686 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.340693    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19af2665-2e6e-4110-9146-fa8ac990d0af-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:55:24.340686 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.340701    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwdhx\" (UniqueName: \"kubernetes.io/projected/19af2665-2e6e-4110-9146-fa8ac990d0af-kube-api-access-pwdhx\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:55:24.340961 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.340710    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19af2665-2e6e-4110-9146-fa8ac990d0af-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:55:24.628480 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.628370    2576 generic.go:356] "Generic (PLEG): container finished" podID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerID="aa00fff6200197e9c1d2efa72ac7bf4966218335ea107a6bcca565dcdb991296" exitCode=0
Jul 28 01:55:24.628480 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.628454    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"
Jul 28 01:55:24.628480 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.628469    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" event={"ID":"19af2665-2e6e-4110-9146-fa8ac990d0af","Type":"ContainerDied","Data":"aa00fff6200197e9c1d2efa72ac7bf4966218335ea107a6bcca565dcdb991296"}
Jul 28 01:55:24.628752 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.628532    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j" event={"ID":"19af2665-2e6e-4110-9146-fa8ac990d0af","Type":"ContainerDied","Data":"694bbeab602ebe5442e7922d150ad7e938d868a258982b1bf80eb49fbd75f40e"}
Jul 28 01:55:24.628752 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.628566    2576 scope.go:117] "RemoveContainer" containerID="e65c73d690174d2e1d3801cd499694ee3a412b7bed65528b97562c24eb55b3bf"
Jul 28 01:55:24.636252 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.636237    2576 scope.go:117] "RemoveContainer" containerID="aa00fff6200197e9c1d2efa72ac7bf4966218335ea107a6bcca565dcdb991296"
Jul 28 01:55:24.643351 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.643334    2576 scope.go:117] "RemoveContainer" containerID="220e0dc6b5c9b864bfd1d659510f8159eddedb125226cdfb1c319738b9d7078e"
Jul 28 01:55:24.645577 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.645555    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"]
Jul 28 01:55:24.648744 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.648726    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-xkv9j"]
Jul 28 01:55:24.651140 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.651123    2576 scope.go:117] "RemoveContainer" containerID="e65c73d690174d2e1d3801cd499694ee3a412b7bed65528b97562c24eb55b3bf"
Jul 28 01:55:24.651429 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:55:24.651407    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65c73d690174d2e1d3801cd499694ee3a412b7bed65528b97562c24eb55b3bf\": container with ID starting with e65c73d690174d2e1d3801cd499694ee3a412b7bed65528b97562c24eb55b3bf not found: ID does not exist" containerID="e65c73d690174d2e1d3801cd499694ee3a412b7bed65528b97562c24eb55b3bf"
Jul 28 01:55:24.651509 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.651438    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65c73d690174d2e1d3801cd499694ee3a412b7bed65528b97562c24eb55b3bf"} err="failed to get container status \"e65c73d690174d2e1d3801cd499694ee3a412b7bed65528b97562c24eb55b3bf\": rpc error: code = NotFound desc = could not find container \"e65c73d690174d2e1d3801cd499694ee3a412b7bed65528b97562c24eb55b3bf\": container with ID starting with e65c73d690174d2e1d3801cd499694ee3a412b7bed65528b97562c24eb55b3bf not found: ID does not exist"
Jul 28 01:55:24.651509 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.651457    2576 scope.go:117] "RemoveContainer" containerID="aa00fff6200197e9c1d2efa72ac7bf4966218335ea107a6bcca565dcdb991296"
Jul 28 01:55:24.651678 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:55:24.651662    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa00fff6200197e9c1d2efa72ac7bf4966218335ea107a6bcca565dcdb991296\": container with ID starting with aa00fff6200197e9c1d2efa72ac7bf4966218335ea107a6bcca565dcdb991296 not found: ID does not exist" containerID="aa00fff6200197e9c1d2efa72ac7bf4966218335ea107a6bcca565dcdb991296"
Jul 28 01:55:24.651725 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.651681    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa00fff6200197e9c1d2efa72ac7bf4966218335ea107a6bcca565dcdb991296"} err="failed to get container status \"aa00fff6200197e9c1d2efa72ac7bf4966218335ea107a6bcca565dcdb991296\": rpc error: code = NotFound desc = could not find container \"aa00fff6200197e9c1d2efa72ac7bf4966218335ea107a6bcca565dcdb991296\": container with ID starting with aa00fff6200197e9c1d2efa72ac7bf4966218335ea107a6bcca565dcdb991296 not found: ID does not exist"
Jul 28 01:55:24.651725 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.651695    2576 scope.go:117] "RemoveContainer" containerID="220e0dc6b5c9b864bfd1d659510f8159eddedb125226cdfb1c319738b9d7078e"
Jul 28 01:55:24.651910 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:55:24.651895    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"220e0dc6b5c9b864bfd1d659510f8159eddedb125226cdfb1c319738b9d7078e\": container with ID starting with 220e0dc6b5c9b864bfd1d659510f8159eddedb125226cdfb1c319738b9d7078e not found: ID does not exist" containerID="220e0dc6b5c9b864bfd1d659510f8159eddedb125226cdfb1c319738b9d7078e"
Jul 28 01:55:24.651968 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:24.651913    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"220e0dc6b5c9b864bfd1d659510f8159eddedb125226cdfb1c319738b9d7078e"} err="failed to get container status \"220e0dc6b5c9b864bfd1d659510f8159eddedb125226cdfb1c319738b9d7078e\": rpc error: code = NotFound desc = could not find container \"220e0dc6b5c9b864bfd1d659510f8159eddedb125226cdfb1c319738b9d7078e\": container with ID starting with 220e0dc6b5c9b864bfd1d659510f8159eddedb125226cdfb1c319738b9d7078e not found: ID does not exist"
Jul 28 01:55:26.430405 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:26.430360    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" path="/var/lib/kubelet/pods/19af2665-2e6e-4110-9146-fa8ac990d0af/volumes"
Jul 28 01:55:29.445065 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.445033    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:29.503668 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503636    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"]
Jul 28 01:55:29.503871 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503859    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kube-rbac-proxy"
Jul 28 01:55:29.503918 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503872    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kube-rbac-proxy"
Jul 28 01:55:29.503918 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503885    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kserve-container"
Jul 28 01:55:29.503918 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503890    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kserve-container"
Jul 28 01:55:29.503918 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503897    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kube-rbac-proxy"
Jul 28 01:55:29.503918 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503903    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kube-rbac-proxy"
Jul 28 01:55:29.503918 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503911    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kserve-container"
Jul 28 01:55:29.503918 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503916    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kserve-container"
Jul 28 01:55:29.504127 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503925    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="storage-initializer"
Jul 28 01:55:29.504127 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503930    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="storage-initializer"
Jul 28 01:55:29.504127 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503939    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="storage-initializer"
Jul 28 01:55:29.504127 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503944    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="storage-initializer"
Jul 28 01:55:29.504127 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503979    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kserve-container"
Jul 28 01:55:29.504127 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503987    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb86c1d-7f2c-456e-806c-dbe14b66b3a3" containerName="kube-rbac-proxy"
Jul 28 01:55:29.504127 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.503994    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kserve-container"
Jul 28 01:55:29.504127 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.504000    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="19af2665-2e6e-4110-9146-fa8ac990d0af" containerName="kube-rbac-proxy"
Jul 28 01:55:29.508435 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.508416    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.510783 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.510765    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 01:55:29.510925 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.510908    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 01:55:29.511000 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.510909    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 01:55:29.511000 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.510964    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-733ea5-predictor-serving-cert\""
Jul 28 01:55:29.511000 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.510978    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 01:55:29.511158 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.511040    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cl94s\""
Jul 28 01:55:29.511158 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.511042    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-733ea5-kube-rbac-proxy-sar-config\""
Jul 28 01:55:29.518429 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.518405    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"]
Jul 28 01:55:29.578141 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.578108    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-cabundle-cert\") pod \"isvc-primary-733ea5-predictor-79b856fbf7-8llcd\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") " pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.578304 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.578160    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-733ea5-predictor-79b856fbf7-8llcd\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") " pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.578304 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.578262    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlfsk\" (UniqueName: \"kubernetes.io/projected/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kube-api-access-dlfsk\") pod \"isvc-primary-733ea5-predictor-79b856fbf7-8llcd\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") " pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.578436 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.578312    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-proxy-tls\") pod \"isvc-primary-733ea5-predictor-79b856fbf7-8llcd\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") " pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.578436 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.578342    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kserve-provision-location\") pod \"isvc-primary-733ea5-predictor-79b856fbf7-8llcd\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") " pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.634075 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.634046    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:29.679634 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.679595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-cabundle-cert\") pod \"isvc-primary-733ea5-predictor-79b856fbf7-8llcd\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") " pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.679634 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.679640    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-733ea5-predictor-79b856fbf7-8llcd\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") " pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.679869 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.679668    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlfsk\" (UniqueName: \"kubernetes.io/projected/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kube-api-access-dlfsk\") pod \"isvc-primary-733ea5-predictor-79b856fbf7-8llcd\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") " pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.679869 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.679698    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-proxy-tls\") pod \"isvc-primary-733ea5-predictor-79b856fbf7-8llcd\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") " pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.679869 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.679725    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kserve-provision-location\") pod \"isvc-primary-733ea5-predictor-79b856fbf7-8llcd\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") " pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.680229 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.680203    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kserve-provision-location\") pod \"isvc-primary-733ea5-predictor-79b856fbf7-8llcd\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") " pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.680426 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.680405    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-cabundle-cert\") pod \"isvc-primary-733ea5-predictor-79b856fbf7-8llcd\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") " pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.680426 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.680403    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-733ea5-predictor-79b856fbf7-8llcd\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") " pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.682261 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.682242    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-proxy-tls\") pod \"isvc-primary-733ea5-predictor-79b856fbf7-8llcd\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") " pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.688798 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.688775    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlfsk\" (UniqueName: \"kubernetes.io/projected/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kube-api-access-dlfsk\") pod \"isvc-primary-733ea5-predictor-79b856fbf7-8llcd\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") " pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.819047 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.819004    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:29.947887 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:29.947852    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"]
Jul 28 01:55:29.951104 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:55:29.951067    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766b16e2_8e42_42a1_9fbd_3f781d4a70d9.slice/crio-3dd1fdcee7c955dcfe060f0e61b965ebcb1100660fbd3f89e8c75075babf1cc8 WatchSource:0}: Error finding container 3dd1fdcee7c955dcfe060f0e61b965ebcb1100660fbd3f89e8c75075babf1cc8: Status 404 returned error can't find the container with id 3dd1fdcee7c955dcfe060f0e61b965ebcb1100660fbd3f89e8c75075babf1cc8
Jul 28 01:55:30.650430 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:30.650378    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" event={"ID":"766b16e2-8e42-42a1-9fbd-3f781d4a70d9","Type":"ContainerStarted","Data":"8081f47f4a2b4ce6ede58a1c71c1bea0f006058d5ae5167ff3a79db745b8798b"}
Jul 28 01:55:30.650854 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:30.650448    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" event={"ID":"766b16e2-8e42-42a1-9fbd-3f781d4a70d9","Type":"ContainerStarted","Data":"3dd1fdcee7c955dcfe060f0e61b965ebcb1100660fbd3f89e8c75075babf1cc8"}
Jul 28 01:55:34.666483 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:34.666440    2576 generic.go:356] "Generic (PLEG): container finished" podID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerID="8081f47f4a2b4ce6ede58a1c71c1bea0f006058d5ae5167ff3a79db745b8798b" exitCode=0
Jul 28 01:55:34.666483 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:34.666488    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" event={"ID":"766b16e2-8e42-42a1-9fbd-3f781d4a70d9","Type":"ContainerDied","Data":"8081f47f4a2b4ce6ede58a1c71c1bea0f006058d5ae5167ff3a79db745b8798b"}
Jul 28 01:55:35.671853 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:35.671814    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" event={"ID":"766b16e2-8e42-42a1-9fbd-3f781d4a70d9","Type":"ContainerStarted","Data":"54900e51feeaa693428414ea8ea15f79076031e34930a5f0d11168dd3cb5baf9"}
Jul 28 01:55:35.672222 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:35.671881    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" event={"ID":"766b16e2-8e42-42a1-9fbd-3f781d4a70d9","Type":"ContainerStarted","Data":"5699f5c2808fae5940d967f95f0c1c9c736ac86fe0e6ef922b842c324cd81288"}
Jul 28 01:55:35.672222 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:35.672080    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:36.675826 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:36.675790    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:36.677159 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:36.677130    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 01:55:37.679363 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:37.679326    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 01:55:42.684227 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:42.684198    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:55:42.684902 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:42.684873    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 01:55:42.703735 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:42.703665    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" podStartSLOduration=10.05109161 podStartE2EDuration="13.703648696s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.652557086s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:55:35.689666474 +0000 UTC m=+1765.860705513" watchObservedRunningTime="2026-07-28 01:55:42.703648696 +0000 UTC m=+1772.874687735"
Jul 28 01:55:52.685683 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:55:52.685647    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 01:56:02.685407 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:56:02.685300    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 01:56:12.685434 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:56:12.685378    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 01:56:22.685266 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:56:22.685222    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 01:56:32.685710 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:56:32.685665    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 01:56:42.685315 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:56:42.685278    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 01:56:48.429979 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:56:48.429952    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:57:04.821297 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:04.821256    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:04.937097 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:04.937053    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:20.738168 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:20.738120    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"]
Jul 28 01:57:20.738697 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:20.738478    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kserve-container" containerID="cri-o://5699f5c2808fae5940d967f95f0c1c9c736ac86fe0e6ef922b842c324cd81288" gracePeriod=30
Jul 28 01:57:20.738697 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:20.738507    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kube-rbac-proxy" containerID="cri-o://54900e51feeaa693428414ea8ea15f79076031e34930a5f0d11168dd3cb5baf9" gracePeriod=30
Jul 28 01:57:20.779962 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:20.779926    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:20.921590 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:20.921557    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:21.031996 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:21.031907    2576 generic.go:356] "Generic (PLEG): container finished" podID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerID="54900e51feeaa693428414ea8ea15f79076031e34930a5f0d11168dd3cb5baf9" exitCode=2
Jul 28 01:57:21.031996 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:21.031951    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" event={"ID":"766b16e2-8e42-42a1-9fbd-3f781d4a70d9","Type":"ContainerDied","Data":"54900e51feeaa693428414ea8ea15f79076031e34930a5f0d11168dd3cb5baf9"}
Jul 28 01:57:22.680270 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:22.680233    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 01:57:25.280913 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.280890    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:57:25.411187 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.411083    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kube-rbac-proxy-sar-config\") pod \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") "
Jul 28 01:57:25.411187 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.411140    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlfsk\" (UniqueName: \"kubernetes.io/projected/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kube-api-access-dlfsk\") pod \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") "
Jul 28 01:57:25.411187 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.411178    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-cabundle-cert\") pod \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") "
Jul 28 01:57:25.411187 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.411195    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kserve-provision-location\") pod \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") "
Jul 28 01:57:25.411581 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.411217    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-proxy-tls\") pod \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\" (UID: \"766b16e2-8e42-42a1-9fbd-3f781d4a70d9\") "
Jul 28 01:57:25.411581 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.411510    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "766b16e2-8e42-42a1-9fbd-3f781d4a70d9" (UID: "766b16e2-8e42-42a1-9fbd-3f781d4a70d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:57:25.411670 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.411577    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "766b16e2-8e42-42a1-9fbd-3f781d4a70d9" (UID: "766b16e2-8e42-42a1-9fbd-3f781d4a70d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:57:25.411670 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.411627    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "766b16e2-8e42-42a1-9fbd-3f781d4a70d9" (UID: "766b16e2-8e42-42a1-9fbd-3f781d4a70d9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:57:25.413495 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.413474    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "766b16e2-8e42-42a1-9fbd-3f781d4a70d9" (UID: "766b16e2-8e42-42a1-9fbd-3f781d4a70d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:57:25.413617 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.413542    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kube-api-access-dlfsk" (OuterVolumeSpecName: "kube-api-access-dlfsk") pod "766b16e2-8e42-42a1-9fbd-3f781d4a70d9" (UID: "766b16e2-8e42-42a1-9fbd-3f781d4a70d9"). InnerVolumeSpecName "kube-api-access-dlfsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:57:25.511925 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.511881    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:57:25.511925 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.511915    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlfsk\" (UniqueName: \"kubernetes.io/projected/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kube-api-access-dlfsk\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:57:25.511925 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.511926    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:57:25.511925 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.511935    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:57:25.512189 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.511944    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/766b16e2-8e42-42a1-9fbd-3f781d4a70d9-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:57:25.972751 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:25.972712    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:26.012980 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.012945    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"]
Jul 28 01:57:26.013202 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.013190    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="storage-initializer"
Jul 28 01:57:26.013249 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.013204    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="storage-initializer"
Jul 28 01:57:26.013249 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.013218    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kube-rbac-proxy"
Jul 28 01:57:26.013249 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.013223    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kube-rbac-proxy"
Jul 28 01:57:26.013249 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.013233    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kserve-container"
Jul 28 01:57:26.013249 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.013237    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kserve-container"
Jul 28 01:57:26.013456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.013275    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kube-rbac-proxy"
Jul 28 01:57:26.013456 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.013284    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerName="kserve-container"
Jul 28 01:57:26.016377 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.016360    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.020157 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.020136    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-40119c\""
Jul 28 01:57:26.021080 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.021065    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-40119c-predictor-serving-cert\""
Jul 28 01:57:26.021198 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.021084    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-40119c-kube-rbac-proxy-sar-config\""
Jul 28 01:57:26.021304 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.021290    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-40119c-dockercfg-hqzv8\""
Jul 28 01:57:26.036819 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.036796    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"]
Jul 28 01:57:26.050107 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.050078    2576 generic.go:356] "Generic (PLEG): container finished" podID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" containerID="5699f5c2808fae5940d967f95f0c1c9c736ac86fe0e6ef922b842c324cd81288" exitCode=0
Jul 28 01:57:26.050222 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.050138    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"
Jul 28 01:57:26.050222 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.050168    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" event={"ID":"766b16e2-8e42-42a1-9fbd-3f781d4a70d9","Type":"ContainerDied","Data":"5699f5c2808fae5940d967f95f0c1c9c736ac86fe0e6ef922b842c324cd81288"}
Jul 28 01:57:26.050315 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.050235    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd" event={"ID":"766b16e2-8e42-42a1-9fbd-3f781d4a70d9","Type":"ContainerDied","Data":"3dd1fdcee7c955dcfe060f0e61b965ebcb1100660fbd3f89e8c75075babf1cc8"}
Jul 28 01:57:26.050315 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.050264    2576 scope.go:117] "RemoveContainer" containerID="54900e51feeaa693428414ea8ea15f79076031e34930a5f0d11168dd3cb5baf9"
Jul 28 01:57:26.058802 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.058787    2576 scope.go:117] "RemoveContainer" containerID="5699f5c2808fae5940d967f95f0c1c9c736ac86fe0e6ef922b842c324cd81288"
Jul 28 01:57:26.065793 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.065777    2576 scope.go:117] "RemoveContainer" containerID="8081f47f4a2b4ce6ede58a1c71c1bea0f006058d5ae5167ff3a79db745b8798b"
Jul 28 01:57:26.072739 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.072724    2576 scope.go:117] "RemoveContainer" containerID="54900e51feeaa693428414ea8ea15f79076031e34930a5f0d11168dd3cb5baf9"
Jul 28 01:57:26.073009 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:57:26.072991    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54900e51feeaa693428414ea8ea15f79076031e34930a5f0d11168dd3cb5baf9\": container with ID starting with 54900e51feeaa693428414ea8ea15f79076031e34930a5f0d11168dd3cb5baf9 not found: ID does not exist" containerID="54900e51feeaa693428414ea8ea15f79076031e34930a5f0d11168dd3cb5baf9"
Jul 28 01:57:26.073057 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.073019    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54900e51feeaa693428414ea8ea15f79076031e34930a5f0d11168dd3cb5baf9"} err="failed to get container status \"54900e51feeaa693428414ea8ea15f79076031e34930a5f0d11168dd3cb5baf9\": rpc error: code = NotFound desc = could not find container \"54900e51feeaa693428414ea8ea15f79076031e34930a5f0d11168dd3cb5baf9\": container with ID starting with 54900e51feeaa693428414ea8ea15f79076031e34930a5f0d11168dd3cb5baf9 not found: ID does not exist"
Jul 28 01:57:26.073057 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.073041    2576 scope.go:117] "RemoveContainer" containerID="5699f5c2808fae5940d967f95f0c1c9c736ac86fe0e6ef922b842c324cd81288"
Jul 28 01:57:26.073291 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:57:26.073275    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5699f5c2808fae5940d967f95f0c1c9c736ac86fe0e6ef922b842c324cd81288\": container with ID starting with 5699f5c2808fae5940d967f95f0c1c9c736ac86fe0e6ef922b842c324cd81288 not found: ID does not exist" containerID="5699f5c2808fae5940d967f95f0c1c9c736ac86fe0e6ef922b842c324cd81288"
Jul 28 01:57:26.073338 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.073297    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5699f5c2808fae5940d967f95f0c1c9c736ac86fe0e6ef922b842c324cd81288"} err="failed to get container status \"5699f5c2808fae5940d967f95f0c1c9c736ac86fe0e6ef922b842c324cd81288\": rpc error: code = NotFound desc = could not find container \"5699f5c2808fae5940d967f95f0c1c9c736ac86fe0e6ef922b842c324cd81288\": container with ID starting with 5699f5c2808fae5940d967f95f0c1c9c736ac86fe0e6ef922b842c324cd81288 not found: ID does not exist"
Jul 28 01:57:26.073338 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.073311    2576 scope.go:117] "RemoveContainer" containerID="8081f47f4a2b4ce6ede58a1c71c1bea0f006058d5ae5167ff3a79db745b8798b"
Jul 28 01:57:26.073550 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:57:26.073532    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8081f47f4a2b4ce6ede58a1c71c1bea0f006058d5ae5167ff3a79db745b8798b\": container with ID starting with 8081f47f4a2b4ce6ede58a1c71c1bea0f006058d5ae5167ff3a79db745b8798b not found: ID does not exist" containerID="8081f47f4a2b4ce6ede58a1c71c1bea0f006058d5ae5167ff3a79db745b8798b"
Jul 28 01:57:26.073594 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.073554    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8081f47f4a2b4ce6ede58a1c71c1bea0f006058d5ae5167ff3a79db745b8798b"} err="failed to get container status \"8081f47f4a2b4ce6ede58a1c71c1bea0f006058d5ae5167ff3a79db745b8798b\": rpc error: code = NotFound desc = could not find container \"8081f47f4a2b4ce6ede58a1c71c1bea0f006058d5ae5167ff3a79db745b8798b\": container with ID starting with 8081f47f4a2b4ce6ede58a1c71c1bea0f006058d5ae5167ff3a79db745b8798b not found: ID does not exist"
Jul 28 01:57:26.095222 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.095197    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"]
Jul 28 01:57:26.099273 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.099251    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-733ea5-predictor-79b856fbf7-8llcd"]
Jul 28 01:57:26.117782 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.117752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84e577f0-d28a-41e1-8595-8156a550a69a-kserve-provision-location\") pod \"isvc-init-fail-40119c-predictor-58899449b-b7mwh\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.117782 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.117792    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8stmr\" (UniqueName: \"kubernetes.io/projected/84e577f0-d28a-41e1-8595-8156a550a69a-kube-api-access-8stmr\") pod \"isvc-init-fail-40119c-predictor-58899449b-b7mwh\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.117998 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.117900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84e577f0-d28a-41e1-8595-8156a550a69a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-40119c-predictor-58899449b-b7mwh\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.117998 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.117947    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84e577f0-d28a-41e1-8595-8156a550a69a-proxy-tls\") pod \"isvc-init-fail-40119c-predictor-58899449b-b7mwh\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.117998 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.117969    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84e577f0-d28a-41e1-8595-8156a550a69a-cabundle-cert\") pod \"isvc-init-fail-40119c-predictor-58899449b-b7mwh\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.218506 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.218467    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84e577f0-d28a-41e1-8595-8156a550a69a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-40119c-predictor-58899449b-b7mwh\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.218706 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.218532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84e577f0-d28a-41e1-8595-8156a550a69a-proxy-tls\") pod \"isvc-init-fail-40119c-predictor-58899449b-b7mwh\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.218706 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.218569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84e577f0-d28a-41e1-8595-8156a550a69a-cabundle-cert\") pod \"isvc-init-fail-40119c-predictor-58899449b-b7mwh\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.218706 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.218602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84e577f0-d28a-41e1-8595-8156a550a69a-kserve-provision-location\") pod \"isvc-init-fail-40119c-predictor-58899449b-b7mwh\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.218706 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.218630    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8stmr\" (UniqueName: \"kubernetes.io/projected/84e577f0-d28a-41e1-8595-8156a550a69a-kube-api-access-8stmr\") pod \"isvc-init-fail-40119c-predictor-58899449b-b7mwh\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.219027 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.219008    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84e577f0-d28a-41e1-8595-8156a550a69a-kserve-provision-location\") pod \"isvc-init-fail-40119c-predictor-58899449b-b7mwh\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.219228 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.219211    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84e577f0-d28a-41e1-8595-8156a550a69a-cabundle-cert\") pod \"isvc-init-fail-40119c-predictor-58899449b-b7mwh\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.219266 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.219211    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84e577f0-d28a-41e1-8595-8156a550a69a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-40119c-predictor-58899449b-b7mwh\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.221430 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.221407    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84e577f0-d28a-41e1-8595-8156a550a69a-proxy-tls\") pod \"isvc-init-fail-40119c-predictor-58899449b-b7mwh\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.228188 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.228167    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8stmr\" (UniqueName: \"kubernetes.io/projected/84e577f0-d28a-41e1-8595-8156a550a69a-kube-api-access-8stmr\") pod \"isvc-init-fail-40119c-predictor-58899449b-b7mwh\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.241473 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.241426    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:26.326417 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.326308    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:26.432312 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.432280    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="766b16e2-8e42-42a1-9fbd-3f781d4a70d9" path="/var/lib/kubelet/pods/766b16e2-8e42-42a1-9fbd-3f781d4a70d9/volumes"
Jul 28 01:57:26.457045 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:26.457006    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"]
Jul 28 01:57:26.459813 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:57:26.459779    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e577f0_d28a_41e1_8595_8156a550a69a.slice/crio-ee5db790dc5fc64b43a8b53c0a493628edcde40f4dba7abbe49750f5d27b6d5e WatchSource:0}: Error finding container ee5db790dc5fc64b43a8b53c0a493628edcde40f4dba7abbe49750f5d27b6d5e: Status 404 returned error can't find the container with id ee5db790dc5fc64b43a8b53c0a493628edcde40f4dba7abbe49750f5d27b6d5e
Jul 28 01:57:27.056944 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:27.056902    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh" event={"ID":"84e577f0-d28a-41e1-8595-8156a550a69a","Type":"ContainerStarted","Data":"5e30e0da8d82810fbec768bdc1c0b5dbe7dfaf582c77220a1321a7019f2bb2dc"}
Jul 28 01:57:27.057114 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:27.056979    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh" event={"ID":"84e577f0-d28a-41e1-8595-8156a550a69a","Type":"ContainerStarted","Data":"ee5db790dc5fc64b43a8b53c0a493628edcde40f4dba7abbe49750f5d27b6d5e"}
Jul 28 01:57:29.064659 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:29.064632    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-40119c-predictor-58899449b-b7mwh_84e577f0-d28a-41e1-8595-8156a550a69a/storage-initializer/0.log"
Jul 28 01:57:29.065014 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:29.064670    2576 generic.go:356] "Generic (PLEG): container finished" podID="84e577f0-d28a-41e1-8595-8156a550a69a" containerID="5e30e0da8d82810fbec768bdc1c0b5dbe7dfaf582c77220a1321a7019f2bb2dc" exitCode=1
Jul 28 01:57:29.065014 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:29.064718    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh" event={"ID":"84e577f0-d28a-41e1-8595-8156a550a69a","Type":"ContainerDied","Data":"5e30e0da8d82810fbec768bdc1c0b5dbe7dfaf582c77220a1321a7019f2bb2dc"}
Jul 28 01:57:30.069906 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:30.069878    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-40119c-predictor-58899449b-b7mwh_84e577f0-d28a-41e1-8595-8156a550a69a/storage-initializer/0.log"
Jul 28 01:57:30.070316 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:30.069981    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh" event={"ID":"84e577f0-d28a-41e1-8595-8156a550a69a","Type":"ContainerStarted","Data":"4b0ec5d1cf24e97d511ad5398c75c30d6b66828b8d790cea6a89ad507d42e42a"}
Jul 28 01:57:31.008822 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:31.008783    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"]
Jul 28 01:57:31.020279 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:31.020244    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:31.074369 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:31.074333    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh" podUID="84e577f0-d28a-41e1-8595-8156a550a69a" containerName="storage-initializer" containerID="cri-o://4b0ec5d1cf24e97d511ad5398c75c30d6b66828b8d790cea6a89ad507d42e42a" gracePeriod=30
Jul 28 01:57:31.107168 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:31.107101    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:33.615628 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.615605    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-40119c-predictor-58899449b-b7mwh_84e577f0-d28a-41e1-8595-8156a550a69a/storage-initializer/1.log"
Jul 28 01:57:33.616002 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.615988    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-40119c-predictor-58899449b-b7mwh_84e577f0-d28a-41e1-8595-8156a550a69a/storage-initializer/0.log"
Jul 28 01:57:33.618473 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.618458    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:33.773352 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.773324    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8stmr\" (UniqueName: \"kubernetes.io/projected/84e577f0-d28a-41e1-8595-8156a550a69a-kube-api-access-8stmr\") pod \"84e577f0-d28a-41e1-8595-8156a550a69a\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") "
Jul 28 01:57:33.773552 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.773361    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84e577f0-d28a-41e1-8595-8156a550a69a-cabundle-cert\") pod \"84e577f0-d28a-41e1-8595-8156a550a69a\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") "
Jul 28 01:57:33.773552 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.773396    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84e577f0-d28a-41e1-8595-8156a550a69a-proxy-tls\") pod \"84e577f0-d28a-41e1-8595-8156a550a69a\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") "
Jul 28 01:57:33.773552 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.773470    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84e577f0-d28a-41e1-8595-8156a550a69a-kube-rbac-proxy-sar-config\") pod \"84e577f0-d28a-41e1-8595-8156a550a69a\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") "
Jul 28 01:57:33.773552 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.773505    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84e577f0-d28a-41e1-8595-8156a550a69a-kserve-provision-location\") pod \"84e577f0-d28a-41e1-8595-8156a550a69a\" (UID: \"84e577f0-d28a-41e1-8595-8156a550a69a\") "
Jul 28 01:57:33.773850 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.773754    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e577f0-d28a-41e1-8595-8156a550a69a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "84e577f0-d28a-41e1-8595-8156a550a69a" (UID: "84e577f0-d28a-41e1-8595-8156a550a69a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:57:33.773850 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.773781    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e577f0-d28a-41e1-8595-8156a550a69a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "84e577f0-d28a-41e1-8595-8156a550a69a" (UID: "84e577f0-d28a-41e1-8595-8156a550a69a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:57:33.773850 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.773815    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e577f0-d28a-41e1-8595-8156a550a69a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "84e577f0-d28a-41e1-8595-8156a550a69a" (UID: "84e577f0-d28a-41e1-8595-8156a550a69a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:57:33.775596 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.775574    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e577f0-d28a-41e1-8595-8156a550a69a-kube-api-access-8stmr" (OuterVolumeSpecName: "kube-api-access-8stmr") pod "84e577f0-d28a-41e1-8595-8156a550a69a" (UID: "84e577f0-d28a-41e1-8595-8156a550a69a"). InnerVolumeSpecName "kube-api-access-8stmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:57:33.775735 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.775714    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e577f0-d28a-41e1-8595-8156a550a69a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "84e577f0-d28a-41e1-8595-8156a550a69a" (UID: "84e577f0-d28a-41e1-8595-8156a550a69a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:57:33.877413 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.875170    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84e577f0-d28a-41e1-8595-8156a550a69a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:57:33.877413 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.875220    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84e577f0-d28a-41e1-8595-8156a550a69a-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:57:33.877413 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.875250    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8stmr\" (UniqueName: \"kubernetes.io/projected/84e577f0-d28a-41e1-8595-8156a550a69a-kube-api-access-8stmr\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:57:33.877413 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.875276    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84e577f0-d28a-41e1-8595-8156a550a69a-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:57:33.877413 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:33.875292    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84e577f0-d28a-41e1-8595-8156a550a69a-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 01:57:34.085615 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:34.085521    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-40119c-predictor-58899449b-b7mwh_84e577f0-d28a-41e1-8595-8156a550a69a/storage-initializer/1.log"
Jul 28 01:57:34.085878 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:34.085863    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-40119c-predictor-58899449b-b7mwh_84e577f0-d28a-41e1-8595-8156a550a69a/storage-initializer/0.log"
Jul 28 01:57:34.085934 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:34.085905    2576 generic.go:356] "Generic (PLEG): container finished" podID="84e577f0-d28a-41e1-8595-8156a550a69a" containerID="4b0ec5d1cf24e97d511ad5398c75c30d6b66828b8d790cea6a89ad507d42e42a" exitCode=1
Jul 28 01:57:34.085977 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:34.085941    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh" event={"ID":"84e577f0-d28a-41e1-8595-8156a550a69a","Type":"ContainerDied","Data":"4b0ec5d1cf24e97d511ad5398c75c30d6b66828b8d790cea6a89ad507d42e42a"}
Jul 28 01:57:34.085977 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:34.085971    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"
Jul 28 01:57:34.086056 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:34.086036    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh" event={"ID":"84e577f0-d28a-41e1-8595-8156a550a69a","Type":"ContainerDied","Data":"ee5db790dc5fc64b43a8b53c0a493628edcde40f4dba7abbe49750f5d27b6d5e"}
Jul 28 01:57:34.086100 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:34.086076    2576 scope.go:117] "RemoveContainer" containerID="4b0ec5d1cf24e97d511ad5398c75c30d6b66828b8d790cea6a89ad507d42e42a"
Jul 28 01:57:34.096505 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:34.096370    2576 scope.go:117] "RemoveContainer" containerID="5e30e0da8d82810fbec768bdc1c0b5dbe7dfaf582c77220a1321a7019f2bb2dc"
Jul 28 01:57:34.103964 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:34.103943    2576 scope.go:117] "RemoveContainer" containerID="4b0ec5d1cf24e97d511ad5398c75c30d6b66828b8d790cea6a89ad507d42e42a"
Jul 28 01:57:34.104239 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:57:34.104220    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b0ec5d1cf24e97d511ad5398c75c30d6b66828b8d790cea6a89ad507d42e42a\": container with ID starting with 4b0ec5d1cf24e97d511ad5398c75c30d6b66828b8d790cea6a89ad507d42e42a not found: ID does not exist" containerID="4b0ec5d1cf24e97d511ad5398c75c30d6b66828b8d790cea6a89ad507d42e42a"
Jul 28 01:57:34.104320 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:34.104251    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b0ec5d1cf24e97d511ad5398c75c30d6b66828b8d790cea6a89ad507d42e42a"} err="failed to get container status \"4b0ec5d1cf24e97d511ad5398c75c30d6b66828b8d790cea6a89ad507d42e42a\": rpc error: code = NotFound desc = could not find container \"4b0ec5d1cf24e97d511ad5398c75c30d6b66828b8d790cea6a89ad507d42e42a\": container with ID starting with 4b0ec5d1cf24e97d511ad5398c75c30d6b66828b8d790cea6a89ad507d42e42a not found: ID does not exist"
Jul 28 01:57:34.104320 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:34.104276    2576 scope.go:117] "RemoveContainer" containerID="5e30e0da8d82810fbec768bdc1c0b5dbe7dfaf582c77220a1321a7019f2bb2dc"
Jul 28 01:57:34.104542 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:57:34.104522    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e30e0da8d82810fbec768bdc1c0b5dbe7dfaf582c77220a1321a7019f2bb2dc\": container with ID starting with 5e30e0da8d82810fbec768bdc1c0b5dbe7dfaf582c77220a1321a7019f2bb2dc not found: ID does not exist" containerID="5e30e0da8d82810fbec768bdc1c0b5dbe7dfaf582c77220a1321a7019f2bb2dc"
Jul 28 01:57:34.104585 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:34.104553    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e30e0da8d82810fbec768bdc1c0b5dbe7dfaf582c77220a1321a7019f2bb2dc"} err="failed to get container status \"5e30e0da8d82810fbec768bdc1c0b5dbe7dfaf582c77220a1321a7019f2bb2dc\": rpc error: code = NotFound desc = could not find container \"5e30e0da8d82810fbec768bdc1c0b5dbe7dfaf582c77220a1321a7019f2bb2dc\": container with ID starting with 5e30e0da8d82810fbec768bdc1c0b5dbe7dfaf582c77220a1321a7019f2bb2dc not found: ID does not exist"
Jul 28 01:57:34.125082 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:34.125055    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"]
Jul 28 01:57:34.129026 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:34.129001    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-40119c-predictor-58899449b-b7mwh"]
Jul 28 01:57:34.430651 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:34.430576    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84e577f0-d28a-41e1-8595-8156a550a69a" path="/var/lib/kubelet/pods/84e577f0-d28a-41e1-8595-8156a550a69a/volumes"
Jul 28 01:57:36.180053 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:36.180012    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:36.575583 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:57:36.575550    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:06.371361 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:06.371271    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:11.496365 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.496329    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:11.561674 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.561643    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"]
Jul 28 01:59:11.561902 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.561889    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e577f0-d28a-41e1-8595-8156a550a69a" containerName="storage-initializer"
Jul 28 01:59:11.561902 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.561903    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e577f0-d28a-41e1-8595-8156a550a69a" containerName="storage-initializer"
Jul 28 01:59:11.562017 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.561961    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="84e577f0-d28a-41e1-8595-8156a550a69a" containerName="storage-initializer"
Jul 28 01:59:11.562017 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.562002    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e577f0-d28a-41e1-8595-8156a550a69a" containerName="storage-initializer"
Jul 28 01:59:11.562017 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.562008    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e577f0-d28a-41e1-8595-8156a550a69a" containerName="storage-initializer"
Jul 28 01:59:11.562111 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.562057    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="84e577f0-d28a-41e1-8595-8156a550a69a" containerName="storage-initializer"
Jul 28 01:59:11.564988 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.564966    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:11.567328 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.567296    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-e7c05-predictor-serving-cert\""
Jul 28 01:59:11.567463 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.567362    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 01:59:11.567532 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.567462    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-e7c05-kube-rbac-proxy-sar-config\""
Jul 28 01:59:11.567532 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.567486    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 01:59:11.567532 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.567471    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 01:59:11.567532 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.567511    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 01:59:11.568140 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.568122    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cl94s\""
Jul 28 01:59:11.574647 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.574623    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"]
Jul 28 01:59:11.655422 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.655363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f98494de-7b4b-4603-81f7-032b92198678-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:11.655585 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.655428    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tq6n\" (UniqueName: \"kubernetes.io/projected/f98494de-7b4b-4603-81f7-032b92198678-kube-api-access-9tq6n\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:11.655585 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.655510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f98494de-7b4b-4603-81f7-032b92198678-cabundle-cert\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:11.655585 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.655560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f98494de-7b4b-4603-81f7-032b92198678-proxy-tls\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:11.655708 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.655607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f98494de-7b4b-4603-81f7-032b92198678-kserve-provision-location\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:11.756741 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.756645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f98494de-7b4b-4603-81f7-032b92198678-cabundle-cert\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:11.756741 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.756697    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f98494de-7b4b-4603-81f7-032b92198678-proxy-tls\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:11.756741 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.756726    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f98494de-7b4b-4603-81f7-032b92198678-kserve-provision-location\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:11.757013 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.756758    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f98494de-7b4b-4603-81f7-032b92198678-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:11.757013 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.756779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tq6n\" (UniqueName: \"kubernetes.io/projected/f98494de-7b4b-4603-81f7-032b92198678-kube-api-access-9tq6n\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:11.757013 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:59:11.756891    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-serving-cert: secret "raw-sklearn-runtime-e7c05-predictor-serving-cert" not found
Jul 28 01:59:11.757013 ip-10-0-128-168 kubenswrapper[2576]: E0728 01:59:11.756967    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98494de-7b4b-4603-81f7-032b92198678-proxy-tls podName:f98494de-7b4b-4603-81f7-032b92198678 nodeName:}" failed. No retries permitted until 2026-07-28 01:59:12.256948248 +0000 UTC m=+1982.427987276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f98494de-7b4b-4603-81f7-032b92198678-proxy-tls") pod "raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" (UID: "f98494de-7b4b-4603-81f7-032b92198678") : secret "raw-sklearn-runtime-e7c05-predictor-serving-cert" not found
Jul 28 01:59:11.757195 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.757170    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f98494de-7b4b-4603-81f7-032b92198678-kserve-provision-location\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:11.757314 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.757296    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f98494de-7b4b-4603-81f7-032b92198678-cabundle-cert\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:11.757367 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.757349    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f98494de-7b4b-4603-81f7-032b92198678-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:11.765243 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.765215    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tq6n\" (UniqueName: \"kubernetes.io/projected/f98494de-7b4b-4603-81f7-032b92198678-kube-api-access-9tq6n\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:11.937194 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:11.937162    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:12.261765 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:12.261731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f98494de-7b4b-4603-81f7-032b92198678-proxy-tls\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:12.264224 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:12.264196    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f98494de-7b4b-4603-81f7-032b92198678-proxy-tls\") pod \"raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:12.475910 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:12.475871    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:12.603108 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:12.603075    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"]
Jul 28 01:59:12.606833 ip-10-0-128-168 kubenswrapper[2576]: W0728 01:59:12.606809    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98494de_7b4b_4603_81f7_032b92198678.slice/crio-8edae7d077e72325368519e325abf88fc8c8f9376f93a99587fb3097860b30c8 WatchSource:0}: Error finding container 8edae7d077e72325368519e325abf88fc8c8f9376f93a99587fb3097860b30c8: Status 404 returned error can't find the container with id 8edae7d077e72325368519e325abf88fc8c8f9376f93a99587fb3097860b30c8
Jul 28 01:59:13.419810 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:13.419775    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" event={"ID":"f98494de-7b4b-4603-81f7-032b92198678","Type":"ContainerStarted","Data":"65f29aadd6c2caf0ecce7d2ca73f6c75673195a6f871165aebbcdf23b3c1e661"}
Jul 28 01:59:13.420008 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:13.419839    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" event={"ID":"f98494de-7b4b-4603-81f7-032b92198678","Type":"ContainerStarted","Data":"8edae7d077e72325368519e325abf88fc8c8f9376f93a99587fb3097860b30c8"}
Jul 28 01:59:16.432604 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:16.432571    2576 generic.go:356] "Generic (PLEG): container finished" podID="f98494de-7b4b-4603-81f7-032b92198678" containerID="65f29aadd6c2caf0ecce7d2ca73f6c75673195a6f871165aebbcdf23b3c1e661" exitCode=0
Jul 28 01:59:16.433000 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:16.432641    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" event={"ID":"f98494de-7b4b-4603-81f7-032b92198678","Type":"ContainerDied","Data":"65f29aadd6c2caf0ecce7d2ca73f6c75673195a6f871165aebbcdf23b3c1e661"}
Jul 28 01:59:17.438629 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:17.438591    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" event={"ID":"f98494de-7b4b-4603-81f7-032b92198678","Type":"ContainerStarted","Data":"87540752edd7a805eb1232a7eb1062ce169290e647c456015360191f181a5dfb"}
Jul 28 01:59:17.439017 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:17.438651    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" event={"ID":"f98494de-7b4b-4603-81f7-032b92198678","Type":"ContainerStarted","Data":"c3af1742394e93dda05fc477d27aa2029407fa0187aa32852adffddad85274a0"}
Jul 28 01:59:17.439017 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:17.438961    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:17.439133 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:17.439112    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:17.440325 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:17.440300    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 01:59:17.459180 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:17.459127    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" podStartSLOduration=2.751411753 podStartE2EDuration="6.45911096s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.707699207s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:59:17.458091585 +0000 UTC m=+1987.629130621" watchObservedRunningTime="2026-07-28 01:59:17.45911096 +0000 UTC m=+1987.630149999"
Jul 28 01:59:18.441960 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:18.441924    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 01:59:23.446637 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:23.446607    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 01:59:23.447218 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:23.447192    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 01:59:33.447257 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:33.447219    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 01:59:43.447665 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:43.447620    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 01:59:53.447152 ip-10-0-128-168 kubenswrapper[2576]: I0728 01:59:53.447108    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 02:00:03.448167 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:03.448127    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 02:00:13.447870 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:13.447829    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 02:00:23.447697 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:23.447661    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 02:00:33.447638 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:33.447555    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 02:00:41.651601 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:41.651534    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"]
Jul 28 02:00:41.652009 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:41.651873    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kserve-container" containerID="cri-o://c3af1742394e93dda05fc477d27aa2029407fa0187aa32852adffddad85274a0" gracePeriod=30
Jul 28 02:00:41.652009 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:41.651931    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kube-rbac-proxy" containerID="cri-o://87540752edd7a805eb1232a7eb1062ce169290e647c456015360191f181a5dfb" gracePeriod=30
Jul 28 02:00:41.676119 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:41.676087    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 02:00:42.724038 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:42.724003    2576 generic.go:356] "Generic (PLEG): container finished" podID="f98494de-7b4b-4603-81f7-032b92198678" containerID="87540752edd7a805eb1232a7eb1062ce169290e647c456015360191f181a5dfb" exitCode=2
Jul 28 02:00:42.724441 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:42.724079    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" event={"ID":"f98494de-7b4b-4603-81f7-032b92198678","Type":"ContainerDied","Data":"87540752edd7a805eb1232a7eb1062ce169290e647c456015360191f181a5dfb"}
Jul 28 02:00:43.442262 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:43.442221    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 02:00:43.447355 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:43.447329    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 02:00:46.395883 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.395859    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 02:00:46.436216 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.436128    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tq6n\" (UniqueName: \"kubernetes.io/projected/f98494de-7b4b-4603-81f7-032b92198678-kube-api-access-9tq6n\") pod \"f98494de-7b4b-4603-81f7-032b92198678\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") "
Jul 28 02:00:46.436216 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.436161    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f98494de-7b4b-4603-81f7-032b92198678-proxy-tls\") pod \"f98494de-7b4b-4603-81f7-032b92198678\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") "
Jul 28 02:00:46.436216 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.436204    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f98494de-7b4b-4603-81f7-032b92198678-kube-rbac-proxy-sar-config\") pod \"f98494de-7b4b-4603-81f7-032b92198678\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") "
Jul 28 02:00:46.436518 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.436307    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f98494de-7b4b-4603-81f7-032b92198678-cabundle-cert\") pod \"f98494de-7b4b-4603-81f7-032b92198678\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") "
Jul 28 02:00:46.436518 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.436350    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f98494de-7b4b-4603-81f7-032b92198678-kserve-provision-location\") pod \"f98494de-7b4b-4603-81f7-032b92198678\" (UID: \"f98494de-7b4b-4603-81f7-032b92198678\") "
Jul 28 02:00:46.436635 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.436602    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98494de-7b4b-4603-81f7-032b92198678-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f98494de-7b4b-4603-81f7-032b92198678" (UID: "f98494de-7b4b-4603-81f7-032b92198678"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:00:46.436716 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.436693    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98494de-7b4b-4603-81f7-032b92198678-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f98494de-7b4b-4603-81f7-032b92198678" (UID: "f98494de-7b4b-4603-81f7-032b92198678"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:00:46.436716 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.436691    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98494de-7b4b-4603-81f7-032b92198678-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f98494de-7b4b-4603-81f7-032b92198678" (UID: "f98494de-7b4b-4603-81f7-032b92198678"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:00:46.438460 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.438439    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98494de-7b4b-4603-81f7-032b92198678-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f98494de-7b4b-4603-81f7-032b92198678" (UID: "f98494de-7b4b-4603-81f7-032b92198678"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:00:46.438576 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.438558    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98494de-7b4b-4603-81f7-032b92198678-kube-api-access-9tq6n" (OuterVolumeSpecName: "kube-api-access-9tq6n") pod "f98494de-7b4b-4603-81f7-032b92198678" (UID: "f98494de-7b4b-4603-81f7-032b92198678"). InnerVolumeSpecName "kube-api-access-9tq6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:00:46.536996 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.536957    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f98494de-7b4b-4603-81f7-032b92198678-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 02:00:46.536996 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.536989    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f98494de-7b4b-4603-81f7-032b92198678-cabundle-cert\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 02:00:46.536996 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.537003    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f98494de-7b4b-4603-81f7-032b92198678-kserve-provision-location\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 02:00:46.537229 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.537015    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tq6n\" (UniqueName: \"kubernetes.io/projected/f98494de-7b4b-4603-81f7-032b92198678-kube-api-access-9tq6n\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 02:00:46.537229 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.537026    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f98494de-7b4b-4603-81f7-032b92198678-proxy-tls\") on node \"ip-10-0-128-168.ec2.internal\" DevicePath \"\""
Jul 28 02:00:46.745285 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.745245    2576 generic.go:356] "Generic (PLEG): container finished" podID="f98494de-7b4b-4603-81f7-032b92198678" containerID="c3af1742394e93dda05fc477d27aa2029407fa0187aa32852adffddad85274a0" exitCode=0
Jul 28 02:00:46.745502 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.745300    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" event={"ID":"f98494de-7b4b-4603-81f7-032b92198678","Type":"ContainerDied","Data":"c3af1742394e93dda05fc477d27aa2029407fa0187aa32852adffddad85274a0"}
Jul 28 02:00:46.745502 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.745318    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"
Jul 28 02:00:46.745502 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.745369    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz" event={"ID":"f98494de-7b4b-4603-81f7-032b92198678","Type":"ContainerDied","Data":"8edae7d077e72325368519e325abf88fc8c8f9376f93a99587fb3097860b30c8"}
Jul 28 02:00:46.745502 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.745416    2576 scope.go:117] "RemoveContainer" containerID="87540752edd7a805eb1232a7eb1062ce169290e647c456015360191f181a5dfb"
Jul 28 02:00:46.753366 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.753347    2576 scope.go:117] "RemoveContainer" containerID="c3af1742394e93dda05fc477d27aa2029407fa0187aa32852adffddad85274a0"
Jul 28 02:00:46.760451 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.760428    2576 scope.go:117] "RemoveContainer" containerID="65f29aadd6c2caf0ecce7d2ca73f6c75673195a6f871165aebbcdf23b3c1e661"
Jul 28 02:00:46.767497 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.767467    2576 scope.go:117] "RemoveContainer" containerID="87540752edd7a805eb1232a7eb1062ce169290e647c456015360191f181a5dfb"
Jul 28 02:00:46.767808 ip-10-0-128-168 kubenswrapper[2576]: E0728 02:00:46.767787    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87540752edd7a805eb1232a7eb1062ce169290e647c456015360191f181a5dfb\": container with ID starting with 87540752edd7a805eb1232a7eb1062ce169290e647c456015360191f181a5dfb not found: ID does not exist" containerID="87540752edd7a805eb1232a7eb1062ce169290e647c456015360191f181a5dfb"
Jul 28 02:00:46.767891 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.767820    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87540752edd7a805eb1232a7eb1062ce169290e647c456015360191f181a5dfb"} err="failed to get container status \"87540752edd7a805eb1232a7eb1062ce169290e647c456015360191f181a5dfb\": rpc error: code = NotFound desc = could not find container \"87540752edd7a805eb1232a7eb1062ce169290e647c456015360191f181a5dfb\": container with ID starting with 87540752edd7a805eb1232a7eb1062ce169290e647c456015360191f181a5dfb not found: ID does not exist"
Jul 28 02:00:46.767891 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.767840    2576 scope.go:117] "RemoveContainer" containerID="c3af1742394e93dda05fc477d27aa2029407fa0187aa32852adffddad85274a0"
Jul 28 02:00:46.768079 ip-10-0-128-168 kubenswrapper[2576]: E0728 02:00:46.768063    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3af1742394e93dda05fc477d27aa2029407fa0187aa32852adffddad85274a0\": container with ID starting with c3af1742394e93dda05fc477d27aa2029407fa0187aa32852adffddad85274a0 not found: ID does not exist" containerID="c3af1742394e93dda05fc477d27aa2029407fa0187aa32852adffddad85274a0"
Jul 28 02:00:46.768130 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.768084    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3af1742394e93dda05fc477d27aa2029407fa0187aa32852adffddad85274a0"} err="failed to get container status \"c3af1742394e93dda05fc477d27aa2029407fa0187aa32852adffddad85274a0\": rpc error: code = NotFound desc = could not find container \"c3af1742394e93dda05fc477d27aa2029407fa0187aa32852adffddad85274a0\": container with ID starting with c3af1742394e93dda05fc477d27aa2029407fa0187aa32852adffddad85274a0 not found: ID does not exist"
Jul 28 02:00:46.768130 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.768098    2576 scope.go:117] "RemoveContainer" containerID="65f29aadd6c2caf0ecce7d2ca73f6c75673195a6f871165aebbcdf23b3c1e661"
Jul 28 02:00:46.768344 ip-10-0-128-168 kubenswrapper[2576]: E0728 02:00:46.768324    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f29aadd6c2caf0ecce7d2ca73f6c75673195a6f871165aebbcdf23b3c1e661\": container with ID starting with 65f29aadd6c2caf0ecce7d2ca73f6c75673195a6f871165aebbcdf23b3c1e661 not found: ID does not exist" containerID="65f29aadd6c2caf0ecce7d2ca73f6c75673195a6f871165aebbcdf23b3c1e661"
Jul 28 02:00:46.768464 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.768349    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f29aadd6c2caf0ecce7d2ca73f6c75673195a6f871165aebbcdf23b3c1e661"} err="failed to get container status \"65f29aadd6c2caf0ecce7d2ca73f6c75673195a6f871165aebbcdf23b3c1e661\": rpc error: code = NotFound desc = could not find container \"65f29aadd6c2caf0ecce7d2ca73f6c75673195a6f871165aebbcdf23b3c1e661\": container with ID starting with 65f29aadd6c2caf0ecce7d2ca73f6c75673195a6f871165aebbcdf23b3c1e661 not found: ID does not exist"
Jul 28 02:00:46.768865 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.768847    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"]
Jul 28 02:00:46.774239 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:46.774218    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-e7c05-predictor-c7b4b4688-j6hlz"]
Jul 28 02:00:47.950000 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:47.949969    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 02:00:48.048170 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:48.048134    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 02:00:48.430946 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:00:48.430917    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f98494de-7b4b-4603-81f7-032b92198678" path="/var/lib/kubelet/pods/f98494de-7b4b-4603-81f7-032b92198678/volumes"
Jul 28 02:01:19.393915 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.393872    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 02:01:19.528174 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.528135    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4jnbh/must-gather-2rvvt"]
Jul 28 02:01:19.528425 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.528410    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kube-rbac-proxy"
Jul 28 02:01:19.528474 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.528428    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kube-rbac-proxy"
Jul 28 02:01:19.528474 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.528444    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kserve-container"
Jul 28 02:01:19.528474 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.528450    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kserve-container"
Jul 28 02:01:19.528474 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.528457    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="storage-initializer"
Jul 28 02:01:19.528474 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.528465    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="storage-initializer"
Jul 28 02:01:19.528630 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.528508    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kube-rbac-proxy"
Jul 28 02:01:19.528630 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.528516    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f98494de-7b4b-4603-81f7-032b92198678" containerName="kserve-container"
Jul 28 02:01:19.531260 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.531243    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jnbh/must-gather-2rvvt"
Jul 28 02:01:19.533714 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.533687    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4jnbh\"/\"kube-root-ca.crt\""
Jul 28 02:01:19.533846 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.533754    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4jnbh\"/\"openshift-service-ca.crt\""
Jul 28 02:01:19.533846 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.533809    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4jnbh\"/\"default-dockercfg-5ljcm\""
Jul 28 02:01:19.539345 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.539325    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jnbh/must-gather-2rvvt"]
Jul 28 02:01:19.545158 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.545135    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 02:01:19.639424 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.639375    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 02:01:19.672923 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.672835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6c2fc153-d34f-4765-a837-6932a9a7b50f-must-gather-output\") pod \"must-gather-2rvvt\" (UID: \"6c2fc153-d34f-4765-a837-6932a9a7b50f\") " pod="openshift-must-gather-4jnbh/must-gather-2rvvt"
Jul 28 02:01:19.672923 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.672870    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7zxd\" (UniqueName: \"kubernetes.io/projected/6c2fc153-d34f-4765-a837-6932a9a7b50f-kube-api-access-v7zxd\") pod \"must-gather-2rvvt\" (UID: \"6c2fc153-d34f-4765-a837-6932a9a7b50f\") " pod="openshift-must-gather-4jnbh/must-gather-2rvvt"
Jul 28 02:01:19.773887 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.773848    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6c2fc153-d34f-4765-a837-6932a9a7b50f-must-gather-output\") pod \"must-gather-2rvvt\" (UID: \"6c2fc153-d34f-4765-a837-6932a9a7b50f\") " pod="openshift-must-gather-4jnbh/must-gather-2rvvt"
Jul 28 02:01:19.773887 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.773882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7zxd\" (UniqueName: \"kubernetes.io/projected/6c2fc153-d34f-4765-a837-6932a9a7b50f-kube-api-access-v7zxd\") pod \"must-gather-2rvvt\" (UID: \"6c2fc153-d34f-4765-a837-6932a9a7b50f\") " pod="openshift-must-gather-4jnbh/must-gather-2rvvt"
Jul 28 02:01:19.774226 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.774209    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6c2fc153-d34f-4765-a837-6932a9a7b50f-must-gather-output\") pod \"must-gather-2rvvt\" (UID: \"6c2fc153-d34f-4765-a837-6932a9a7b50f\") " pod="openshift-must-gather-4jnbh/must-gather-2rvvt"
Jul 28 02:01:19.782262 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.782228    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7zxd\" (UniqueName: \"kubernetes.io/projected/6c2fc153-d34f-4765-a837-6932a9a7b50f-kube-api-access-v7zxd\") pod \"must-gather-2rvvt\" (UID: \"6c2fc153-d34f-4765-a837-6932a9a7b50f\") " pod="openshift-must-gather-4jnbh/must-gather-2rvvt"
Jul 28 02:01:19.840999 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.840958    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jnbh/must-gather-2rvvt"
Jul 28 02:01:19.969562 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.969522    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jnbh/must-gather-2rvvt"]
Jul 28 02:01:19.974355 ip-10-0-128-168 kubenswrapper[2576]: W0728 02:01:19.974320    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c2fc153_d34f_4765_a837_6932a9a7b50f.slice/crio-898ca8a88c34c4176e80996fd722becfc6be800685718fdf091598aa463cbcb8 WatchSource:0}: Error finding container 898ca8a88c34c4176e80996fd722becfc6be800685718fdf091598aa463cbcb8: Status 404 returned error can't find the container with id 898ca8a88c34c4176e80996fd722becfc6be800685718fdf091598aa463cbcb8
Jul 28 02:01:19.976276 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:19.976259    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 02:01:20.857552 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:20.857376    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jnbh/must-gather-2rvvt" event={"ID":"6c2fc153-d34f-4765-a837-6932a9a7b50f","Type":"ContainerStarted","Data":"898ca8a88c34c4176e80996fd722becfc6be800685718fdf091598aa463cbcb8"}
Jul 28 02:01:21.866257 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:21.866215    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jnbh/must-gather-2rvvt" event={"ID":"6c2fc153-d34f-4765-a837-6932a9a7b50f","Type":"ContainerStarted","Data":"11fd22a9464a54ea3e83b975aceca4b2d666f597b841e50d057244a304c55ed0"}
Jul 28 02:01:21.866804 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:21.866275    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jnbh/must-gather-2rvvt" event={"ID":"6c2fc153-d34f-4765-a837-6932a9a7b50f","Type":"ContainerStarted","Data":"aa80d3a494664a48b657aee561ce50e510012c83a3d0cdf7368b13efe431559f"}
Jul 28 02:01:21.881988 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:21.881925    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4jnbh/must-gather-2rvvt" podStartSLOduration=1.9399603669999999 podStartE2EDuration="2.881904016s" totalImagesPullingTime="941.943649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:01:21.881327647 +0000 UTC m=+2112.052366685" watchObservedRunningTime="2026-07-28 02:01:21.881904016 +0000 UTC m=+2112.052943054"
Jul 28 02:01:22.343039 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:22.342971    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nf7cb_be103d49-24c3-4a2e-9a55-2fcc960c1a71/global-pull-secret-syncer/0.log"
Jul 28 02:01:22.397202 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:22.397172    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log"
Jul 28 02:01:22.479437 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:22.479411    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-168.ec2.internal_088df6a04689e2bf1edce0757bca5189/haproxy/0.log"
Jul 28 02:01:25.117500 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:25.117437    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 02:01:25.190516 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:25.190481    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 02:01:27.007399 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:27.007351    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cngcv_297c5f42-e728-4ee6-92c9-ac2cefeb2fa0/node-exporter/0.log"
Jul 28 02:01:27.042543 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:27.042509    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cngcv_297c5f42-e728-4ee6-92c9-ac2cefeb2fa0/kube-rbac-proxy/0.log"
Jul 28 02:01:27.068496 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:27.068469    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cngcv_297c5f42-e728-4ee6-92c9-ac2cefeb2fa0/init-textfile/0.log"
Jul 28 02:01:29.997800 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:29.997762    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cdqxp_3aa211a5-d538-4fd8-a415-29ea6de08067/konnectivity-agent/0.log" limit=1024
Jul 28 02:01:30.676446 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.676411    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"]
Jul 28 02:01:30.679531 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.679501    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.690017 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.689980    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"]
Jul 28 02:01:30.767277 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.767242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/85cb296e-28f8-489d-b748-d19ca3ea3e81-proc\") pod \"perf-node-gather-daemonset-5mqs7\" (UID: \"85cb296e-28f8-489d-b748-d19ca3ea3e81\") " pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.767568 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.767551    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/85cb296e-28f8-489d-b748-d19ca3ea3e81-podres\") pod \"perf-node-gather-daemonset-5mqs7\" (UID: \"85cb296e-28f8-489d-b748-d19ca3ea3e81\") " pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.767715 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.767701    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85cb296e-28f8-489d-b748-d19ca3ea3e81-sys\") pod \"perf-node-gather-daemonset-5mqs7\" (UID: \"85cb296e-28f8-489d-b748-d19ca3ea3e81\") " pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.767830 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.767816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85cb296e-28f8-489d-b748-d19ca3ea3e81-lib-modules\") pod \"perf-node-gather-daemonset-5mqs7\" (UID: \"85cb296e-28f8-489d-b748-d19ca3ea3e81\") " pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.767945 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.767931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlwwl\" (UniqueName: \"kubernetes.io/projected/85cb296e-28f8-489d-b748-d19ca3ea3e81-kube-api-access-zlwwl\") pod \"perf-node-gather-daemonset-5mqs7\" (UID: \"85cb296e-28f8-489d-b748-d19ca3ea3e81\") " pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.868860 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.868816    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/85cb296e-28f8-489d-b748-d19ca3ea3e81-podres\") pod \"perf-node-gather-daemonset-5mqs7\" (UID: \"85cb296e-28f8-489d-b748-d19ca3ea3e81\") " pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.869049 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.868895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85cb296e-28f8-489d-b748-d19ca3ea3e81-sys\") pod \"perf-node-gather-daemonset-5mqs7\" (UID: \"85cb296e-28f8-489d-b748-d19ca3ea3e81\") " pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.869049 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.868937    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85cb296e-28f8-489d-b748-d19ca3ea3e81-lib-modules\") pod \"perf-node-gather-daemonset-5mqs7\" (UID: \"85cb296e-28f8-489d-b748-d19ca3ea3e81\") " pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.869049 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.868982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlwwl\" (UniqueName: \"kubernetes.io/projected/85cb296e-28f8-489d-b748-d19ca3ea3e81-kube-api-access-zlwwl\") pod \"perf-node-gather-daemonset-5mqs7\" (UID: \"85cb296e-28f8-489d-b748-d19ca3ea3e81\") " pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.869049 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.869021    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/85cb296e-28f8-489d-b748-d19ca3ea3e81-proc\") pod \"perf-node-gather-daemonset-5mqs7\" (UID: \"85cb296e-28f8-489d-b748-d19ca3ea3e81\") " pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.869242 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.869126    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/85cb296e-28f8-489d-b748-d19ca3ea3e81-proc\") pod \"perf-node-gather-daemonset-5mqs7\" (UID: \"85cb296e-28f8-489d-b748-d19ca3ea3e81\") " pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.869295 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.869244    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/85cb296e-28f8-489d-b748-d19ca3ea3e81-podres\") pod \"perf-node-gather-daemonset-5mqs7\" (UID: \"85cb296e-28f8-489d-b748-d19ca3ea3e81\") " pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.869344 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.869296    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85cb296e-28f8-489d-b748-d19ca3ea3e81-sys\") pod \"perf-node-gather-daemonset-5mqs7\" (UID: \"85cb296e-28f8-489d-b748-d19ca3ea3e81\") " pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.869413 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.869375    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85cb296e-28f8-489d-b748-d19ca3ea3e81-lib-modules\") pod \"perf-node-gather-daemonset-5mqs7\" (UID: \"85cb296e-28f8-489d-b748-d19ca3ea3e81\") " pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.880293 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.880225    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlwwl\" (UniqueName: \"kubernetes.io/projected/85cb296e-28f8-489d-b748-d19ca3ea3e81-kube-api-access-zlwwl\") pod \"perf-node-gather-daemonset-5mqs7\" (UID: \"85cb296e-28f8-489d-b748-d19ca3ea3e81\") " pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:30.993775 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:30.993738    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:31.154660 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:31.154625    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"]
Jul 28 02:01:31.158342 ip-10-0-128-168 kubenswrapper[2576]: W0728 02:01:31.158308    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod85cb296e_28f8_489d_b748_d19ca3ea3e81.slice/crio-0ab6f91aee209006f7593b39f3444ed6fc6ec0364388ebb9900bd44e68abeab8 WatchSource:0}: Error finding container 0ab6f91aee209006f7593b39f3444ed6fc6ec0364388ebb9900bd44e68abeab8: Status 404 returned error can't find the container with id 0ab6f91aee209006f7593b39f3444ed6fc6ec0364388ebb9900bd44e68abeab8
Jul 28 02:01:31.752775 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:31.752739    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cj2zl_2960b4e1-5587-486f-95b3-aa18155fd6fe/dns/0.log"
Jul 28 02:01:31.775296 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:31.775271    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cj2zl_2960b4e1-5587-486f-95b3-aa18155fd6fe/kube-rbac-proxy/0.log"
Jul 28 02:01:31.912728 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:31.912693    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7" event={"ID":"85cb296e-28f8-489d-b748-d19ca3ea3e81","Type":"ContainerStarted","Data":"21432d4b0f3a116abbc2979b582630efcb86f323559f1239883baa7ff07da4dd"}
Jul 28 02:01:31.912936 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:31.912754    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7" event={"ID":"85cb296e-28f8-489d-b748-d19ca3ea3e81","Type":"ContainerStarted","Data":"0ab6f91aee209006f7593b39f3444ed6fc6ec0364388ebb9900bd44e68abeab8"}
Jul 28 02:01:31.912936 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:31.912798    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:31.916528 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:31.916507    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jbm5x_bc143eaf-873e-484b-9ef0-adc12e645c43/dns-node-resolver/0.log"
Jul 28 02:01:32.443818 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:32.443787    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q2vqt_45a5d7f9-5b74-4f84-a448-d89cb7445b23/node-ca/0.log"
Jul 28 02:01:33.540811 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:33.540783    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6zpwr_f01ecde2-601f-4327-b1d6-43b8b46fa2d5/serve-healthcheck-canary/0.log"
Jul 28 02:01:33.914577 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:33.914507    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5vv8g_a78712d7-20fe-49b8-94c9-96ac63bd9015/kube-rbac-proxy/0.log"
Jul 28 02:01:33.934821 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:33.934791    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5vv8g_a78712d7-20fe-49b8-94c9-96ac63bd9015/exporter/0.log"
Jul 28 02:01:33.956792 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:33.956768    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5vv8g_a78712d7-20fe-49b8-94c9-96ac63bd9015/extractor/0.log"
Jul 28 02:01:36.306021 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:36.305981    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-ff96d5959-nzsqn_be0d01d3-4102-4a71-81eb-30fc23f7a68f/manager/0.log"
Jul 28 02:01:36.337208 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:36.337177    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84fc959947-tmxdx_adc41827-89af-4789-81c2-30c4f420f89d/manager/0.log"
Jul 28 02:01:37.925686 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:37.925660    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7"
Jul 28 02:01:37.941615 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:37.941568    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4jnbh/perf-node-gather-daemonset-5mqs7" podStartSLOduration=7.941553006 podStartE2EDuration="7.941553006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 02:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:01:31.931770365 +0000 UTC m=+2122.102809403" watchObservedRunningTime="2026-07-28 02:01:37.941553006 +0000 UTC m=+2128.112592042"
Jul 28 02:01:40.658057 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:40.658021    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-hq8wj_b199680c-03b6-4cff-b96b-2aeddfd47867/migrator/0.log"
Jul 28 02:01:40.682987 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:40.682957    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-hq8wj_b199680c-03b6-4cff-b96b-2aeddfd47867/graceful-termination/0.log"
Jul 28 02:01:41.924796 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:41.924769    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2h5rf_89eecf33-ed8e-41fe-8fce-48a72361f9ad/kube-multus/0.log"
Jul 28 02:01:42.136586 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:42.136563    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kwgbx_4db16d2d-9789-4546-96e8-3438dd7fdba8/kube-multus-additional-cni-plugins/0.log"
Jul 28 02:01:42.164064 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:42.163897    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kwgbx_4db16d2d-9789-4546-96e8-3438dd7fdba8/egress-router-binary-copy/0.log"
Jul 28 02:01:42.186676 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:42.186647    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kwgbx_4db16d2d-9789-4546-96e8-3438dd7fdba8/cni-plugins/0.log"
Jul 28 02:01:42.208184 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:42.208152    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kwgbx_4db16d2d-9789-4546-96e8-3438dd7fdba8/bond-cni-plugin/0.log"
Jul 28 02:01:42.232489 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:42.232464    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kwgbx_4db16d2d-9789-4546-96e8-3438dd7fdba8/routeoverride-cni/0.log"
Jul 28 02:01:42.259794 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:42.259767    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kwgbx_4db16d2d-9789-4546-96e8-3438dd7fdba8/whereabouts-cni-bincopy/0.log"
Jul 28 02:01:42.281208 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:42.281181    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kwgbx_4db16d2d-9789-4546-96e8-3438dd7fdba8/whereabouts-cni/0.log"
Jul 28 02:01:42.523145 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:42.523117    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-22fhh_8849435f-9f9d-4fd2-858c-b037077d839c/network-metrics-daemon/0.log"
Jul 28 02:01:42.544311 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:42.544279    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-22fhh_8849435f-9f9d-4fd2-858c-b037077d839c/kube-rbac-proxy/0.log"
Jul 28 02:01:43.605114 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:43.605082    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8dgm_61c607d9-7b10-4b44-88ee-e81ada7ef179/ovn-controller/0.log"
Jul 28 02:01:43.623826 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:43.623801    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8dgm_61c607d9-7b10-4b44-88ee-e81ada7ef179/ovn-acl-logging/0.log"
Jul 28 02:01:43.646800 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:43.646774    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8dgm_61c607d9-7b10-4b44-88ee-e81ada7ef179/kube-rbac-proxy-node/0.log"
Jul 28 02:01:43.668282 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:43.668258    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8dgm_61c607d9-7b10-4b44-88ee-e81ada7ef179/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 02:01:43.692864 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:43.692831    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8dgm_61c607d9-7b10-4b44-88ee-e81ada7ef179/northd/0.log"
Jul 28 02:01:43.714833 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:43.714805    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8dgm_61c607d9-7b10-4b44-88ee-e81ada7ef179/nbdb/0.log"
Jul 28 02:01:43.737004 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:43.736974    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8dgm_61c607d9-7b10-4b44-88ee-e81ada7ef179/sbdb/0.log"
Jul 28 02:01:43.780985 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:43.780957    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8dgm_61c607d9-7b10-4b44-88ee-e81ada7ef179/ovnkube-controller/0.log"
Jul 28 02:01:45.061206 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:45.061164    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-km9pg_14745d0d-a327-4289-84d7-213c1e2f3eef/network-check-target-container/0.log"
Jul 28 02:01:45.909542 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:45.909516    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-p4bgp_d356f315-756f-439b-882e-35320cae4a10/iptables-alerter/0.log"
Jul 28 02:01:46.634963 ip-10-0-128-168 kubenswrapper[2576]: I0728 02:01:46.634933    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-jh4fs_08fb009b-b1f6-465f-b258-7e408d59d0d4/tuned/0.log"