Jul 31 23:46:03.349996 ip-10-0-137-252 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 23:46:03.728535 ip-10-0-137-252 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:46:03.728535 ip-10-0-137-252 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 23:46:03.728535 ip-10-0-137-252 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:46:03.728535 ip-10-0-137-252 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 23:46:03.728535 ip-10-0-137-252 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:46:03.730228 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.730140    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 23:46:03.733590 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733573    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:03.733590 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733590    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733594    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733597    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733601    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733603    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733606    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733609    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733611    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733614    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733617    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733620    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733623    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733626    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733628    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733631    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733634    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733636    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733639    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733641    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733651    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:03.733657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733654    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733656    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733659    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733662    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733665    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733667    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733670    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733673    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733676    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733678    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733681    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733683    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733686    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733688    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733691    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733693    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733696    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733699    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733703    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:03.734170 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733707    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733710    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733713    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733726    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733729    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733732    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733736    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733739    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733742    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733745    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733748    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733750    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733755    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733758    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733761    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733765    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733768    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733771    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733774    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:03.734633 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733777    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733780    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733782    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733785    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733788    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733790    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733793    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733795    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733798    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733800    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733803    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733806    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733809    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733812    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733814    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733817    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733819    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733822    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733824    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733827    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:03.735116 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733830    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733832    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733835    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733838    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733840    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733843    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733847    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733851    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733854    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733857    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733859    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733862    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733864    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733867    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733869    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733872    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733874    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733877    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733879    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.733882    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:03.735594 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735412    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735420    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735423    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735426    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735429    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735432    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735435    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735438    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735441    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735443    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735446    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735448    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735453    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735456    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735459    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735461    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735464    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735466    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735469    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735473    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:03.736109 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735475    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735478    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735480    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735483    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735486    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735488    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735491    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735493    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735496    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735499    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735501    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735504    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735506    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735509    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735511    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735514    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735517    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735520    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735524    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:03.736606 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735526    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735529    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735531    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735534    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735536    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735539    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735541    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735544    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735546    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735549    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735551    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735554    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735556    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735559    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735561    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735564    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735566    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735569    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735571    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735574    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:03.737097 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735576    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735579    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735581    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735584    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735586    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735589    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735591    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735594    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735596    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735601    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735604    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735606    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735610    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735613    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735615    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735618    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735620    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735623    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735625    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735628    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:03.737595 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735631    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735634    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735636    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735639    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735644    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735648    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735650    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735653    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735656    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735659    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735662    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735664    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735667    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735672    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735675    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735677    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735680    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735683    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735686    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:03.738136 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.735690    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736141    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736147    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736151    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736154    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736157    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736160    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736163    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736166    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736168    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736171    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736174    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736176    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736179    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736182    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736184    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736187    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736190    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736194    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:03.738608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736198    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736201    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736204    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736207    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736210    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736213    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736216    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736219    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736222    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736224    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736227    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736230    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736232    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736236    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736240    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736242    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736245    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736248    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736250    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736253    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:03.739096 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736255    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736258    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736261    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736264    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736267    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736270    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736273    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736276    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736279    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736282    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736284    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736287    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736289    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736292    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736295    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736297    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736300    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736303    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736305    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736308    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:03.739609 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736311    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736314    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736316    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736319    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736321    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736324    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736327    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736329    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736332    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736335    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736337    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736340    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736343    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736345    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736348    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736350    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736353    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736355    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736358    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:03.740121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736360    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736364    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736367    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736370    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736373    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736376    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736379    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736381    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736384    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736386    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736389    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736391    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736394    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736396    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736398    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736401    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736403    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736406    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736408    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:03.740563 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736411    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736413    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.736416    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736489    2584 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736502    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736509    2584 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736514    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736518    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736522    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736526    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736531    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736534    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736537    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736541    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736544    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736547    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736550    2584 flags.go:64] FLAG: --cgroup-root=""
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736553    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736556    2584 flags.go:64] FLAG: --client-ca-file=""
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736558    2584 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736561    2584 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736566    2584 flags.go:64] FLAG: --cluster-domain=""
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736568    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736571    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 23:46:03.741065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736575    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736578    2584 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736582    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736585    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736588    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736591    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736594    2584 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736597    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736600    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736603    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736606    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736610    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736613    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736616    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736619    2584 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736622    2584 flags.go:64] FLAG: --enable-server="true"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736625    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736633    2584 flags.go:64] FLAG: --event-burst="100"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736636    2584 flags.go:64] FLAG: --event-qps="50"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736639    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736642    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736645    2584 flags.go:64] FLAG: --eviction-hard=""
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736648    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736651    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736654    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 23:46:03.741646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736657    2584 flags.go:64] FLAG: --eviction-soft=""
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736660    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736663    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736666    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736669    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736672    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736675    2584 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736677    2584 flags.go:64] FLAG: --feature-gates=""
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736681    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736684    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736687    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736690    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736693    2584 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736696    2584 flags.go:64] FLAG: --help="false"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736699    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-137-252.ec2.internal"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736702    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736705    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736707    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736710    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736726    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736729    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736732    2584 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736735    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 23:46:03.742274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736738    2584 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736742    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736746    2584 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736749    2584 flags.go:64] FLAG: --kube-reserved=""
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736752    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736755    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736758    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736761    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736764    2584 flags.go:64] FLAG: --lock-file=""
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736766    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736770    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736773    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736779    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736782    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736785    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736788    2584 flags.go:64] FLAG: --logging-format="text"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736790    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736793    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736796    2584 flags.go:64] FLAG: --manifest-url=""
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736799    2584 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736804    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736807    2584 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736811    2584 flags.go:64] FLAG: --max-pods="110"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736814    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736817    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 23:46:03.742895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736819    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736823    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736826    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736829    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736832    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736838    2584 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736841    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736846    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736850    2584 flags.go:64] FLAG: --pod-cidr=""
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736854    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736860    2584 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736863    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736866    2584 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736869    2584 flags.go:64] FLAG: --port="10250"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736871    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736874    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-097c4e20b97c0fa8d"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736877    2584 flags.go:64] FLAG: --qos-reserved=""
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736881    2584 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736883    2584 flags.go:64] FLAG: --register-node="true"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736887    2584 flags.go:64] FLAG: --register-with-taints=""
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736890    2584 flags.go:64] FLAG: --registry-burst="10"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736893    2584 flags.go:64] FLAG: --registry-qps="5"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736896    2584 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736899    2584 flags.go:64] FLAG: --reserved-memory=""
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736903    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 23:46:03.743828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736906    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736909    2584 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736912    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736915    2584 flags.go:64] FLAG: --runonce="false"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736918    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736921    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736924    2584 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736927    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736930    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736933    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736936    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736939    2584 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736942    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736945    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736948    2584 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736953    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736956    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736959    2584 flags.go:64] FLAG: --system-cgroups=""
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736964    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736969    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736972    2584 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736975    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736979    2584 flags.go:64] FLAG: --tls-min-version=""
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736982    2584 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736985    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 23:46:03.744987 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736987    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736990    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736993    2584 flags.go:64] FLAG: --v="2"
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.736997    2584 flags.go:64] FLAG: --version="false"
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.737001    2584 flags.go:64] FLAG: --vmodule=""
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.737005    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.737008    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737122    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737126    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737130    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737132    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737135    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737138    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737141    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737143    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737146    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737149    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737151    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737154    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737157    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737159    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:03.746065 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737162    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737167    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737170    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737173    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737176    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737180    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737183    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737186    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737189    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737192    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737194    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737197    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737200    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737204    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737207    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737211    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737214    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737216    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737219    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:03.747163 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737222    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737224    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737227    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737229    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737232    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737234    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737237    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737239    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737242    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737244    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737247    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737249    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737252    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737254    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737259    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737262    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737264    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737267    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737271    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737273    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:03.747769 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737276    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737278    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737281    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737283    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737286    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737288    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737291    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737293    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737296    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737298    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737301    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737303    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737305    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737308    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737311    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737313    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737317    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737320    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737323    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:03.748281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737325    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737327    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737330    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737333    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737335    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737338    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737340    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737346    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737349    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737351    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737354    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737357    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737359    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737362    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737365    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737367    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737370    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737372    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737375    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737378    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:03.748770 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737380    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:03.749273 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737383    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:03.749273 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737385    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:03.749273 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737388    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:03.749273 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737390    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:03.749273 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737393    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:03.749273 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.737395    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:03.749273 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.737880    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:46:03.749273 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.746059    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 23:46:03.749273 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.746083    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 23:46:03.749273 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.746109    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 23:46:03.749273 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.746118    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 23:46:03.749273 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746605    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:03.749273 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746616    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:03.749273 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746622    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:03.749273 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746629    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746634    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746646    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746651    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746655    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746660    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746666    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746673    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746681    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746686    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746691    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746695    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746700    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746704    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746731    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746736    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746741    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746745    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746748    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746752    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:03.749657 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746762    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746766    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746773    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746777    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746781    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746786    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746825    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746837    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746842    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746847    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746852    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746859    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746867    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746872    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746877    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746882    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746887    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746891    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746896    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746906    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:03.750184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746911    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746915    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746920    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746926    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746930    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746935    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746939    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746944    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746949    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746953    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746958    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746963    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746973    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746978    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746982    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746986    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746991    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.746996    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747002    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747038    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:03.750736 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747044    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747049    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747054    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747058    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747063    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747072    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747077    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747081    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747086    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747092    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747096    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747101    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747106    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747110    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747114    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747119    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747123    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747133    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747138    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:03.751248 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747142    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747147    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747152    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747156    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747161    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747165    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747169    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747173    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747177    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747182    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747186    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747197    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747202    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747238    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747246    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747251    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:03.751845 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747256    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:03.752281 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.747267    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:46:03.752281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747697    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:03.752281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747710    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:03.752281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747730    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:03.752281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747736    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:03.752281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747739    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:03.752281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747742    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:03.752281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747746    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:03.752281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747749    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:03.752281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747752    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:03.752281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747754    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:03.752281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747757    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:03.752281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747760    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:03.752281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747764    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:03.752281 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747766    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747769    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747771    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747774    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747777    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747780    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747782    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747785    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747788    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747791    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747793    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747796    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747798    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747801    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747804    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747808    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747811    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747813    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747815    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747818    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:03.752926 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747821    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747824    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747827    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747830    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747832    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747835    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747839    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747844    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747847    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747850    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747854    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747858    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747861    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747864    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747867    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747870    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747873    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747876    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747878    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:03.753451 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747881    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747883    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747886    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747889    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747892    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747895    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747898    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747900    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747909    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747913    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747915    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747918    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747921    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747924    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747927    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747930    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747932    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747935    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747938    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747941    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:03.754165 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747943    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747947    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747950    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747954    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747957    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747959    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747962    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747964    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747967    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747969    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747972    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747975    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747977    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747980    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747983    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747985    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747988    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747990    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747993    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:03.754790 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747996    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:03.755253 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.747998    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:03.755253 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.748001    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:03.755253 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.748004    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:03.755253 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.748006    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:03.755253 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.748009    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:03.755253 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.748012    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:03.755253 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:03.748015    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:03.755253 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.748020    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:46:03.755253 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.748176    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 23:46:03.755253 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.750060    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 23:46:03.755253 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.751401    2584 server.go:1019] "Starting client certificate rotation"
Jul 31 23:46:03.755253 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.751508    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 23:46:03.755253 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.751557    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 23:46:03.755637 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.753574    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 23:46:03.756446 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.756417    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 23:46:03.768379 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.768362    2584 log.go:25] "Validated CRI v1 runtime API"
Jul 31 23:46:03.770544 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.770530    2584 log.go:25] "Validated CRI v1 image API"
Jul 31 23:46:03.772055 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.772039    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 23:46:03.774349 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.774329    2584 fs.go:135] Filesystem UUIDs: map[76e4e711-d42f-463c-975d-bac7ca1e9e87:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cf8e8e41-7e45-4b85-8f27-210803edb3e6:/dev/nvme0n1p3]
Jul 31 23:46:03.774402 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.774351    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 23:46:03.780057 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.780040    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 23:46:03.780911 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.780791    2584 manager.go:216] Machine: {Timestamp:2026-07-31 23:46:03.779496273 +0000 UTC m=+0.317385823 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100152 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26fb1240f474599dce9497087a8c3b SystemUUID:ec26fb12-40f4-7459-9dce-9497087a8c3b BootID:f8cb8dce-17da-48a4-8fdd-20b041c4a450 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f1:1a:c7:1f:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f1:1a:c7:1f:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:40:c5:61:31:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 23:46:03.780911 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.780912    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 23:46:03.781054 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.781043    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 23:46:03.783989 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.783966    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 23:46:03.784143 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.783991    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-252.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 23:46:03.784684 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.784674    2584 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 23:46:03.784713 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.784687    2584 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 23:46:03.784713 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.784694    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 23:46:03.785397 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.785387    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 23:46:03.786116 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.786106    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 23:46:03.789629 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.789617    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 23:46:03.789770 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.789760    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 23:46:03.789807 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.789793    2584 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 23:46:03.789846 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.789817    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 23:46:03.789846 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.789827    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 23:46:03.789846 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.789837    2584 kubelet.go:398] "Adding apiserver pod source"
Jul 31 23:46:03.789971 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.789897    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 23:46:03.790602 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.790580    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 23:46:03.791341 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.791311    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 23:46:03.794390 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.794372    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 23:46:03.795851 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.795837    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 23:46:03.795918 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.795867    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 23:46:03.796820 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.796808    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 23:46:03.796863 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.796824    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 23:46:03.796863 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.796830    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 23:46:03.796863 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.796836    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 23:46:03.796863 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.796845    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 23:46:03.796863 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.796850    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 23:46:03.796863 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.796858    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 23:46:03.796863 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.796863    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 23:46:03.797034 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.796870    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 23:46:03.797034 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.796877    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 23:46:03.797034 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.796885    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 23:46:03.797034 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.796890    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 23:46:03.798057 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.798046    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 23:46:03.798057 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.798057    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 23:46:03.801085 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.801066    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-252.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 23:46:03.801327 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.801316    2584 server.go:1268] "Started kubelet"
Jul 31 23:46:03.801451 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.801396    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 23:46:03.801599 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.801540    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 23:46:03.801651 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.801612    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 23:46:03.801693 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:03.801624    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 23:46:03.801693 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:03.801646    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 23:46:03.802149 ip-10-0-137-252 systemd[1]: Started Kubernetes Kubelet.
Jul 31 23:46:03.802839 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.802774    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 23:46:03.803935 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.803919    2584 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 23:46:03.807035 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:03.807008    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 23:46:03.807869 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:03.806659    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-252.ec2.internal.18c7849792123e01  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-252.ec2.internal,UID:ip-10-0-137-252.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-252.ec2.internal,},FirstTimestamp:2026-07-31 23:46:03.801288193 +0000 UTC m=+0.339177743,LastTimestamp:2026-07-31 23:46:03.801288193 +0000 UTC m=+0.339177743,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-252.ec2.internal,}"
Jul 31 23:46:03.808543 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.808527    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 23:46:03.808592 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.808533    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 23:46:03.808958 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:03.808698    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:03.809093 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.809077    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 23:46:03.809093 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.809085    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 23:46:03.809210 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.809110    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 23:46:03.809318 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.809302    2584 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 23:46:03.809318 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.809314    2584 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 23:46:03.809523 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.809493    2584 factory.go:55] Registering systemd factory
Jul 31 23:46:03.809523 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.809511    2584 factory.go:223] Registration of the systemd container factory successfully
Jul 31 23:46:03.809754 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.809737    2584 factory.go:153] Registering CRI-O factory
Jul 31 23:46:03.809754 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.809756    2584 factory.go:223] Registration of the crio container factory successfully
Jul 31 23:46:03.809915 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.809842    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 23:46:03.809915 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.809865    2584 factory.go:103] Registering Raw factory
Jul 31 23:46:03.809915 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.809878    2584 manager.go:1196] Started watching for new ooms in manager
Jul 31 23:46:03.810446 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.810432    2584 manager.go:349] Starting recovery of all containers
Jul 31 23:46:03.822265 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:03.822239    2584 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 23:46:03.822381 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:03.822213    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 23:46:03.824873 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.824849    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zfn9h"
Jul 31 23:46:03.825113 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.825098    2584 manager.go:354] Recovery completed
Jul 31 23:46:03.829812 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.829797    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:03.831621 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.831604    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:03.831682 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.831638    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:03.831682 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.831651    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:03.832069 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.832044    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zfn9h"
Jul 31 23:46:03.832237 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.832224    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 23:46:03.832237 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.832233    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:03.832362 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.832238    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 23:46:03.832362 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.832257    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 23:46:03.832935 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.832919    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:03.832981 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.832948    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:03.832981 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.832958    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:03.834409 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.834396    2584 policy_none.go:49] "None policy: Start"
Jul 31 23:46:03.834487 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.834413    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 23:46:03.834487 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.834426    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 23:46:03.835599 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.835586    2584 policy_none.go:47] "Start"
Jul 31 23:46:03.872255 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.872058    2584 manager.go:337] "Starting Device Plugin manager"
Jul 31 23:46:03.872350 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:03.872278    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 23:46:03.872350 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.872293    2584 server.go:87] "Starting device plugin registration server"
Jul 31 23:46:03.872601 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.872589    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 23:46:03.872637 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.872603    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 23:46:03.872777 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.872763    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 23:46:03.872865 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.872852    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 23:46:03.872916 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.872867    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 23:46:03.873393 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:03.873373    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 23:46:03.873483 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:03.873426    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:03.899229 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.899200    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 23:46:03.902892 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.900514    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 23:46:03.902892 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.900555    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 23:46:03.902892 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.900586    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 23:46:03.902892 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:03.900625    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 23:46:03.902892 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.902764    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:03.973561 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.973515    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:03.974483 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.974460    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:03.974562 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.974494    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:03.974562 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.974505    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:03.974562 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:03.974529    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-252.ec2.internal"
Jul 31 23:46:05.647568 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.647552    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-252.ec2.internal"
Jul 31 23:46:05.647631 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:05.647577    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-252.ec2.internal\": node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:05.665132 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.665097    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-252.ec2.internal"]
Jul 31 23:46:05.665249 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.665185    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:05.666775 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.666759    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:05.666828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.666789    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:05.666828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.666798    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:05.669051 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.669038    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:05.669200 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.669186    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal"
Jul 31 23:46:05.669246 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.669218    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:05.669840 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.669821    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:05.669908 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.669851    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:05.669908 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.669865    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:05.669908 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.669865    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:05.669908 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.669891    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:05.669908 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.669907    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:05.671236 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.671213    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-252.ec2.internal"
Jul 31 23:46:05.671315 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.671246    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:05.671997 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.671983    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:05.672082 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.672029    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:05.672082 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.672060    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:05.674877 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.674859    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f0884df17fa2ea19d188a1590c26af3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal\" (UID: \"4f0884df17fa2ea19d188a1590c26af3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal"
Jul 31 23:46:05.674978 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.674883    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f0884df17fa2ea19d188a1590c26af3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal\" (UID: \"4f0884df17fa2ea19d188a1590c26af3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal"
Jul 31 23:46:05.674978 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.674905    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e413b343bfdc4c93b43b646e37497322-config\") pod \"kube-apiserver-proxy-ip-10-0-137-252.ec2.internal\" (UID: \"e413b343bfdc4c93b43b646e37497322\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-252.ec2.internal"
Jul 31 23:46:05.693347 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:05.693318    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:05.694760 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:05.694744    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-252.ec2.internal\" not found" node="ip-10-0-137-252.ec2.internal"
Jul 31 23:46:05.699325 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:05.699308    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-252.ec2.internal\" not found" node="ip-10-0-137-252.ec2.internal"
Jul 31 23:46:05.775825 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.775785    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e413b343bfdc4c93b43b646e37497322-config\") pod \"kube-apiserver-proxy-ip-10-0-137-252.ec2.internal\" (UID: \"e413b343bfdc4c93b43b646e37497322\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-252.ec2.internal"
Jul 31 23:46:05.775825 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.775824    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f0884df17fa2ea19d188a1590c26af3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal\" (UID: \"4f0884df17fa2ea19d188a1590c26af3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal"
Jul 31 23:46:05.776062 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.775847    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f0884df17fa2ea19d188a1590c26af3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal\" (UID: \"4f0884df17fa2ea19d188a1590c26af3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal"
Jul 31 23:46:05.776062 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.775877    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f0884df17fa2ea19d188a1590c26af3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal\" (UID: \"4f0884df17fa2ea19d188a1590c26af3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal"
Jul 31 23:46:05.776062 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.775878    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e413b343bfdc4c93b43b646e37497322-config\") pod \"kube-apiserver-proxy-ip-10-0-137-252.ec2.internal\" (UID: \"e413b343bfdc4c93b43b646e37497322\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-252.ec2.internal"
Jul 31 23:46:05.776062 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.775888    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f0884df17fa2ea19d188a1590c26af3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal\" (UID: \"4f0884df17fa2ea19d188a1590c26af3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal"
Jul 31 23:46:05.793915 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:05.793894    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:05.894429 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:05.894405    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:05.995262 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:05.995196    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:05.997374 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:05.997357    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal"
Jul 31 23:46:06.002168 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:06.002147    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-252.ec2.internal"
Jul 31 23:46:06.096114 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:06.096076    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:06.196663 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:06.196634    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:06.297168 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:06.297089    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:06.397715 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:06.397685    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:06.414873 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:06.414849    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 23:46:06.415039 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:06.415023    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 23:46:06.498708 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:06.498675    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:06.499776 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:06.499749    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 23:41:05 +0000 UTC" deadline="2028-04-12 09:26:07.615683158 +0000 UTC"
Jul 31 23:46:06.499829 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:06.499777    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14889h40m1.115909297s"
Jul 31 23:46:06.505189 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:06.505166    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:06.531364 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:06.531334    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0884df17fa2ea19d188a1590c26af3.slice/crio-aee9f0727f1ab0c338669f776c6dd2691123e2d42a10d4edbda31f1d9b64091c WatchSource:0}: Error finding container aee9f0727f1ab0c338669f776c6dd2691123e2d42a10d4edbda31f1d9b64091c: Status 404 returned error can't find the container with id aee9f0727f1ab0c338669f776c6dd2691123e2d42a10d4edbda31f1d9b64091c
Jul 31 23:46:06.531597 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:06.531578    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode413b343bfdc4c93b43b646e37497322.slice/crio-882cdd5db5d79bfcf4f743d631d05743d375e7e8e666ceb6053e0ee39f69314f WatchSource:0}: Error finding container 882cdd5db5d79bfcf4f743d631d05743d375e7e8e666ceb6053e0ee39f69314f: Status 404 returned error can't find the container with id 882cdd5db5d79bfcf4f743d631d05743d375e7e8e666ceb6053e0ee39f69314f
Jul 31 23:46:06.535478 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:06.535464    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:46:06.568925 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:06.568829    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal" event={"ID":"4f0884df17fa2ea19d188a1590c26af3","Type":"ContainerStarted","Data":"aee9f0727f1ab0c338669f776c6dd2691123e2d42a10d4edbda31f1d9b64091c"}
Jul 31 23:46:06.569918 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:06.569881    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-252.ec2.internal" event={"ID":"e413b343bfdc4c93b43b646e37497322","Type":"ContainerStarted","Data":"882cdd5db5d79bfcf4f743d631d05743d375e7e8e666ceb6053e0ee39f69314f"}
Jul 31 23:46:06.599336 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:06.599304    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:06.675342 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:06.675319    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:06.699824 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:06.699800    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:06.800298 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:06.800263    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:06.900813 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:06.900738    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-252.ec2.internal\" not found"
Jul 31 23:46:06.961646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:06.961609    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:06.973835 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:06.973803    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal"
Jul 31 23:46:06.988184 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:06.988158    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 23:46:06.989142 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:06.989115    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-252.ec2.internal"
Jul 31 23:46:06.998148 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:06.998120    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 23:46:07.456126 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.456092    2584 apiserver.go:52] "Watching apiserver"
Jul 31 23:46:07.464746 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.464717    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 23:46:07.465804 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.465776    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-qpmwv","openshift-ovn-kubernetes/ovnkube-node-khzpq","openshift-cluster-node-tuning-operator/tuned-q7xzv","openshift-dns/node-resolver-hsjmv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal","openshift-multus/multus-7hjxn","openshift-multus/network-metrics-daemon-hsnnq","openshift-network-operator/iptables-alerter-c6n68","kube-system/global-pull-secret-syncer-sjj4w","kube-system/konnectivity-agent-dhhpx","kube-system/kube-apiserver-proxy-ip-10-0-137-252.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w","openshift-image-registry/node-ca-s9w8v","openshift-multus/multus-additional-cni-plugins-glccn"]
Jul 31 23:46:07.467959 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.467939    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:07.468073 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:07.468046    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:07.469356 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.469337    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.470861 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.470845    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.472131 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.472114    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 23:46:07.472645 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.472625    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 23:46:07.472847 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.472834    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 23:46:07.473143 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.473125    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j86rj\""
Jul 31 23:46:07.473405 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.473388    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:07.473710 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.473693    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:07.473870 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.473855    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9mk2s\""
Jul 31 23:46:07.474055 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.474039    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hsjmv"
Jul 31 23:46:07.474512 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.474495    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 23:46:07.474599 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.474583    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.475126 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.475108    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.476386 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.476365    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 23:46:07.476674 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.476655    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xs4q6\""
Jul 31 23:46:07.476780 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.476721    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 23:46:07.477210 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.477189    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 23:46:07.477415 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.477384    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 23:46:07.477489 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.477462    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pbqsg\""
Jul 31 23:46:07.477556 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.477529    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c6n68"
Jul 31 23:46:07.477785 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.477763    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 23:46:07.477886 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.477848    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 23:46:07.478069 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.478050    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 23:46:07.478170 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.478155    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kf2cg\""
Jul 31 23:46:07.478238 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.478172    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 23:46:07.478879 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.478859    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s9w8v"
Jul 31 23:46:07.480476 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.480061    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:07.480476 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.480098    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:07.480476 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.480382    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 23:46:07.480476 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.480403    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jsrzq\""
Jul 31 23:46:07.480476 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.480409    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.481397 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.481376    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 23:46:07.481897 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.481877    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 23:46:07.482348 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.482328    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 23:46:07.482763 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.482745    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dhhpx"
Jul 31 23:46:07.483669 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.482861    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-shzh7\""
Jul 31 23:46:07.483669 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.483444    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bmfrb\""
Jul 31 23:46:07.483933 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.483911    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 23:46:07.484472 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.484417    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 23:46:07.484472 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.484456    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 23:46:07.484764 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.484744    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:07.484881 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:07.484861    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:07.485009 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.484638    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 23:46:07.485936 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.485882    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1df3a7f4-80f0-4a56-a857-d59531130023-hosts-file\") pod \"node-resolver-hsjmv\" (UID: \"1df3a7f4-80f0-4a56-a857-d59531130023\") " pod="openshift-dns/node-resolver-hsjmv"
Jul 31 23:46:07.485936 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.485919    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.486101 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.485945    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-run-netns\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.486101 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486007    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-var-lib-cni-multus\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.486101 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486032    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-etc-selinux\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.486101 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486055    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-modprobe-d\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.486101 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486078    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-systemd\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.486303 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486102    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-host\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.486303 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486128    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3079ae51-76cf-4bde-90bf-05e1396256b3-tmp\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.486303 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486152    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z972r\" (UniqueName: \"kubernetes.io/projected/3079ae51-76cf-4bde-90bf-05e1396256b3-kube-api-access-z972r\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.486303 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486186    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-cni-binary-copy\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.486303 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486208    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-kubelet-dir\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.486303 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486231    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-registration-dir\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.486303 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486254    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-sysctl-d\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.486303 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486285    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-var-lib-kubelet\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486328    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-tuned\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486353    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pmpt\" (UniqueName: \"kubernetes.io/projected/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-kube-api-access-9pmpt\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486379    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-multus-socket-dir-parent\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486401    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-var-lib-cni-bin\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486425    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m8p9\" (UniqueName: \"kubernetes.io/projected/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-kube-api-access-6m8p9\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486453    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zn56\" (UniqueName: \"kubernetes.io/projected/1df3a7f4-80f0-4a56-a857-d59531130023-kube-api-access-8zn56\") pod \"node-resolver-hsjmv\" (UID: \"1df3a7f4-80f0-4a56-a857-d59531130023\") " pod="openshift-dns/node-resolver-hsjmv"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486477    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs\") pod \"network-metrics-daemon-hsnnq\" (UID: \"9d3952c7-cede-4f37-aba3-9b96b84eaeb6\") " pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486501    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-device-dir\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486521    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-sys-fs\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486536    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgdc7\" (UniqueName: \"kubernetes.io/projected/984d46fc-6501-4bbb-8b1f-c3bd36b84718-kube-api-access-pgdc7\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486551    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-run\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486567    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-lib-modules\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486566    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486580    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1df3a7f4-80f0-4a56-a857-d59531130023-tmp-dir\") pod \"node-resolver-hsjmv\" (UID: \"1df3a7f4-80f0-4a56-a857-d59531130023\") " pod="openshift-dns/node-resolver-hsjmv"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486600    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-cnibin\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486640    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.486654 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486660    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-system-cni-dir\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486685    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-multus-cni-dir\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486704    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486717    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-cnibin\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486743    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-run-k8s-cni-cncf-io\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486761    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-var-lib-kubelet\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486783    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-hostroot\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486803    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486851    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486429    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486805    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-multus-daemon-config\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:07.486944    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486996    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-etc-kubernetes\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.487031    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-sysconfig\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.487056    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-system-cni-dir\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.487070    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wl482\""
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.487132    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.487136    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-os-release\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.487165    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-multus-conf-dir\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.486188    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 23:46:07.487525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.487201    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-run-multus-certs\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.488535 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.487233    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-sys\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.488535 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.487263    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-os-release\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.488535 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.487288    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdftg\" (UniqueName: \"kubernetes.io/projected/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-kube-api-access-cdftg\") pod \"network-metrics-daemon-hsnnq\" (UID: \"9d3952c7-cede-4f37-aba3-9b96b84eaeb6\") " pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:07.488535 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.487310    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-socket-dir\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.488535 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.487335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.488535 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.487359    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-kubernetes\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.488535 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.487380    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-sysctl-conf\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.508354 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.508334    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4c4tg"
Jul 31 23:46:07.516189 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.516168    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4c4tg"
Jul 31 23:46:07.525458 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.525430    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:07.574245 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.574220    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 23:46:07.588384 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588353    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.588384 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588389    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-multus-cni-dir\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.588566 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588414    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-var-lib-kubelet\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.588566 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588473    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-var-lib-kubelet\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.588566 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588555    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-multus-daemon-config\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.588708 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588587    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-run-ovn\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.588708 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588605    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-cnibin\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.588708 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588619    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-hostroot\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.588708 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588635    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-etc-kubernetes\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.588708 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588647    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-multus-cni-dir\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.588708 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588652    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-run-netns\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.588708 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588687    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-cni-netd\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588708    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-hostroot\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588711    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-cnibin\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588713    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52e351d7-6ba9-4265-bf15-4fcc3604a693-ovn-node-metrics-cert\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588777    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce48bcb1-ccef-4183-af51-8a88c043cb64-serviceca\") pod \"node-ca-s9w8v\" (UID: \"ce48bcb1-ccef-4183-af51-8a88c043cb64\") " pod="openshift-image-registry/node-ca-s9w8v"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588712    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-etc-kubernetes\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588807    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6vx7\" (UniqueName: \"kubernetes.io/projected/ce48bcb1-ccef-4183-af51-8a88c043cb64-kube-api-access-l6vx7\") pod \"node-ca-s9w8v\" (UID: \"ce48bcb1-ccef-4183-af51-8a88c043cb64\") " pod="openshift-image-registry/node-ca-s9w8v"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588840    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-sysconfig\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588864    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-os-release\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588888    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-multus-conf-dir\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588895    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-sysconfig\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588919    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b717d75a-de98-4e7b-b86e-1af68be8f6a7-dbus\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588926    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-multus-conf-dir\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588943    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-sys\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588951    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-os-release\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.588986    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-os-release\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589011    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d23c6534-149f-4a0b-baea-af7e105c571c-host-slash\") pod \"iptables-alerter-c6n68\" (UID: \"d23c6534-149f-4a0b-baea-af7e105c571c\") " pod="openshift-network-operator/iptables-alerter-c6n68"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589021    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-sys\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.589072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589049    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-kubernetes\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589072    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-sysctl-conf\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589077    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-os-release\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589111    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589119    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-kubernetes\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589139    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-etc-selinux\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589165    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-slash\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589188    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2f6w\" (UniqueName: \"kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w\") pod \"network-check-target-qpmwv\" (UID: \"6ca51531-7776-413a-98b3-aebadd923730\") " pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589197    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-multus-daemon-config\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589210    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-sysctl-conf\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589213    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-modprobe-d\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589255    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589260    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-host\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589281    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3079ae51-76cf-4bde-90bf-05e1396256b3-tmp\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589304    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-var-lib-kubelet\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589310    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-modprobe-d\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589313    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-host\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.589860 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589326    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmpt\" (UniqueName: \"kubernetes.io/projected/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-kube-api-access-9pmpt\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589344    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-etc-selinux\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589357    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-var-lib-cni-bin\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589379    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-var-lib-kubelet\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589384    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m8p9\" (UniqueName: \"kubernetes.io/projected/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-kube-api-access-6m8p9\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589412    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-kubelet\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589437    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-run-ovn-kubernetes\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589461    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-cni-bin\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589473    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-var-lib-cni-bin\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589482    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52e351d7-6ba9-4265-bf15-4fcc3604a693-env-overrides\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs\") pod \"network-metrics-daemon-hsnnq\" (UID: \"9d3952c7-cede-4f37-aba3-9b96b84eaeb6\") " pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589536    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-systemd-units\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589552    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-node-log\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589576    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce48bcb1-ccef-4183-af51-8a88c043cb64-host\") pod \"node-ca-s9w8v\" (UID: \"ce48bcb1-ccef-4183-af51-8a88c043cb64\") " pod="openshift-image-registry/node-ca-s9w8v"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589595    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-run\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589625    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1df3a7f4-80f0-4a56-a857-d59531130023-tmp-dir\") pod \"node-resolver-hsjmv\" (UID: \"1df3a7f4-80f0-4a56-a857-d59531130023\") " pod="openshift-dns/node-resolver-hsjmv"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589640    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-system-cni-dir\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:07.589641    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:07.590507 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589602    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589686    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-run\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589730    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-run-k8s-cni-cncf-io\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589760    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:07.589797    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs podName:9d3952c7-cede-4f37-aba3-9b96b84eaeb6 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:08.089758431 +0000 UTC m=+2.963402022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs") pod "network-metrics-daemon-hsnnq" (UID: "9d3952c7-cede-4f37-aba3-9b96b84eaeb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589829    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589857    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52e351d7-6ba9-4265-bf15-4fcc3604a693-ovnkube-script-lib\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589797    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-system-cni-dir\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589916    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-run-k8s-cni-cncf-io\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589931    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qncgn\" (UniqueName: \"kubernetes.io/projected/d23c6534-149f-4a0b-baea-af7e105c571c-kube-api-access-qncgn\") pod \"iptables-alerter-c6n68\" (UID: \"d23c6534-149f-4a0b-baea-af7e105c571c\") " pod="openshift-network-operator/iptables-alerter-c6n68"
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589990    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-system-cni-dir\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590018    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-run-multus-certs\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590026    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1df3a7f4-80f0-4a56-a857-d59531130023-tmp-dir\") pod \"node-resolver-hsjmv\" (UID: \"1df3a7f4-80f0-4a56-a857-d59531130023\") " pod="openshift-dns/node-resolver-hsjmv"
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590062    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-system-cni-dir\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590097    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-run-multus-certs\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590100    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-etc-openvswitch\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.591224 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590151    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-log-socket\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590180    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/776b1ef2-f435-44ca-9f0e-90316affeef8-konnectivity-ca\") pod \"konnectivity-agent-dhhpx\" (UID: \"776b1ef2-f435-44ca-9f0e-90316affeef8\") " pod="kube-system/konnectivity-agent-dhhpx"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590211    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590246    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdftg\" (UniqueName: \"kubernetes.io/projected/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-kube-api-access-cdftg\") pod \"network-metrics-daemon-hsnnq\" (UID: \"9d3952c7-cede-4f37-aba3-9b96b84eaeb6\") " pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590249    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.589812    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590276    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-socket-dir\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590316    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-run-systemd\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590340    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d23c6534-149f-4a0b-baea-af7e105c571c-iptables-alerter-script\") pod \"iptables-alerter-c6n68\" (UID: \"d23c6534-149f-4a0b-baea-af7e105c571c\") " pod="openshift-network-operator/iptables-alerter-c6n68"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590366    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-sys-fs\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590384    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-socket-dir\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590391    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590418    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1df3a7f4-80f0-4a56-a857-d59531130023-hosts-file\") pod \"node-resolver-hsjmv\" (UID: \"1df3a7f4-80f0-4a56-a857-d59531130023\") " pod="openshift-dns/node-resolver-hsjmv"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590440    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-run-netns\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590447    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-sys-fs\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590463    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-var-lib-cni-multus\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590484    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1df3a7f4-80f0-4a56-a857-d59531130023-hosts-file\") pod \"node-resolver-hsjmv\" (UID: \"1df3a7f4-80f0-4a56-a857-d59531130023\") " pod="openshift-dns/node-resolver-hsjmv"
Jul 31 23:46:07.591903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590491    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52e351d7-6ba9-4265-bf15-4fcc3604a693-ovnkube-config\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590494    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-run-netns\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590515    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-host-var-lib-cni-multus\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590518    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-systemd\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590553    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z972r\" (UniqueName: \"kubernetes.io/projected/3079ae51-76cf-4bde-90bf-05e1396256b3-kube-api-access-z972r\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590557    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-systemd\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590569    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-cni-binary-copy\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590583    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-kubelet-dir\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590601    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-registration-dir\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590651    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-registration-dir\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590653    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-kubelet-dir\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590747    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-var-lib-openvswitch\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590777    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-run-openvswitch\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590800    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-sysctl-d\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590824    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-tuned\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590848    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-multus-socket-dir-parent\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590861    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.592558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590870    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zg94\" (UniqueName: \"kubernetes.io/projected/52e351d7-6ba9-4265-bf15-4fcc3604a693-kube-api-access-9zg94\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.593137 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590893    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/776b1ef2-f435-44ca-9f0e-90316affeef8-agent-certs\") pod \"konnectivity-agent-dhhpx\" (UID: \"776b1ef2-f435-44ca-9f0e-90316affeef8\") " pod="kube-system/konnectivity-agent-dhhpx"
Jul 31 23:46:07.593137 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590900    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-sysctl-d\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.593137 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590915    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b717d75a-de98-4e7b-b86e-1af68be8f6a7-kubelet-config\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:07.593137 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590933    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-multus-socket-dir-parent\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.593137 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.590948    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zn56\" (UniqueName: \"kubernetes.io/projected/1df3a7f4-80f0-4a56-a857-d59531130023-kube-api-access-8zn56\") pod \"node-resolver-hsjmv\" (UID: \"1df3a7f4-80f0-4a56-a857-d59531130023\") " pod="openshift-dns/node-resolver-hsjmv"
Jul 31 23:46:07.593137 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.591004    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-device-dir\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.593137 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.591027    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgdc7\" (UniqueName: \"kubernetes.io/projected/984d46fc-6501-4bbb-8b1f-c3bd36b84718-kube-api-access-pgdc7\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.593137 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.591054    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-lib-modules\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.593137 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.591076    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-cnibin\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.593137 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.591117    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-cni-binary-copy\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.593137 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.591179    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/984d46fc-6501-4bbb-8b1f-c3bd36b84718-device-dir\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.593137 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.591187    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-cnibin\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.593137 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.591315    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3079ae51-76cf-4bde-90bf-05e1396256b3-lib-modules\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.593570 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.593181    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3079ae51-76cf-4bde-90bf-05e1396256b3-tmp\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.593570 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.593207    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3079ae51-76cf-4bde-90bf-05e1396256b3-etc-tuned\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.599566 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.599541    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pmpt\" (UniqueName: \"kubernetes.io/projected/054662f6-b4fc-47bf-afbc-dee3eec7a1a7-kube-api-access-9pmpt\") pod \"multus-additional-cni-plugins-glccn\" (UID: \"054662f6-b4fc-47bf-afbc-dee3eec7a1a7\") " pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.599726 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.599547    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m8p9\" (UniqueName: \"kubernetes.io/projected/3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc-kube-api-access-6m8p9\") pod \"multus-7hjxn\" (UID: \"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc\") " pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.602561 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.602227    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zn56\" (UniqueName: \"kubernetes.io/projected/1df3a7f4-80f0-4a56-a857-d59531130023-kube-api-access-8zn56\") pod \"node-resolver-hsjmv\" (UID: \"1df3a7f4-80f0-4a56-a857-d59531130023\") " pod="openshift-dns/node-resolver-hsjmv"
Jul 31 23:46:07.602561 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.602414    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgdc7\" (UniqueName: \"kubernetes.io/projected/984d46fc-6501-4bbb-8b1f-c3bd36b84718-kube-api-access-pgdc7\") pod \"aws-ebs-csi-driver-node-85q4w\" (UID: \"984d46fc-6501-4bbb-8b1f-c3bd36b84718\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.603087 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.603062    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdftg\" (UniqueName: \"kubernetes.io/projected/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-kube-api-access-cdftg\") pod \"network-metrics-daemon-hsnnq\" (UID: \"9d3952c7-cede-4f37-aba3-9b96b84eaeb6\") " pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:07.603552 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.603531    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z972r\" (UniqueName: \"kubernetes.io/projected/3079ae51-76cf-4bde-90bf-05e1396256b3-kube-api-access-z972r\") pod \"tuned-q7xzv\" (UID: \"3079ae51-76cf-4bde-90bf-05e1396256b3\") " pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.691765 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.691726    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zg94\" (UniqueName: \"kubernetes.io/projected/52e351d7-6ba9-4265-bf15-4fcc3604a693-kube-api-access-9zg94\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.691765 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.691755    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/776b1ef2-f435-44ca-9f0e-90316affeef8-agent-certs\") pod \"konnectivity-agent-dhhpx\" (UID: \"776b1ef2-f435-44ca-9f0e-90316affeef8\") " pod="kube-system/konnectivity-agent-dhhpx"
Jul 31 23:46:07.691765 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.691771    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b717d75a-de98-4e7b-b86e-1af68be8f6a7-kubelet-config\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:07.692057 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.691801    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-run-ovn\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692057 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.691829    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-run-netns\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692057 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.691852    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-cni-netd\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692057 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.691853    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b717d75a-de98-4e7b-b86e-1af68be8f6a7-kubelet-config\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:07.692057 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.691893    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52e351d7-6ba9-4265-bf15-4fcc3604a693-ovn-node-metrics-cert\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692057 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.691899    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-run-netns\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692057 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.691913    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-run-ovn\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692057 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.691928    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce48bcb1-ccef-4183-af51-8a88c043cb64-serviceca\") pod \"node-ca-s9w8v\" (UID: \"ce48bcb1-ccef-4183-af51-8a88c043cb64\") " pod="openshift-image-registry/node-ca-s9w8v"
Jul 31 23:46:07.692057 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.691981    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6vx7\" (UniqueName: \"kubernetes.io/projected/ce48bcb1-ccef-4183-af51-8a88c043cb64-kube-api-access-l6vx7\") pod \"node-ca-s9w8v\" (UID: \"ce48bcb1-ccef-4183-af51-8a88c043cb64\") " pod="openshift-image-registry/node-ca-s9w8v"
Jul 31 23:46:07.692057 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.691995    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-cni-netd\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692057 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692011    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b717d75a-de98-4e7b-b86e-1af68be8f6a7-dbus\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692073    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b717d75a-de98-4e7b-b86e-1af68be8f6a7-dbus\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692100    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d23c6534-149f-4a0b-baea-af7e105c571c-host-slash\") pod \"iptables-alerter-c6n68\" (UID: \"d23c6534-149f-4a0b-baea-af7e105c571c\") " pod="openshift-network-operator/iptables-alerter-c6n68"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692125    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-slash\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692171    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-slash\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692182    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d23c6534-149f-4a0b-baea-af7e105c571c-host-slash\") pod \"iptables-alerter-c6n68\" (UID: \"d23c6534-149f-4a0b-baea-af7e105c571c\") " pod="openshift-network-operator/iptables-alerter-c6n68"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692216    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2f6w\" (UniqueName: \"kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w\") pod \"network-check-target-qpmwv\" (UID: \"6ca51531-7776-413a-98b3-aebadd923730\") " pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692265    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-kubelet\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692295    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-run-ovn-kubernetes\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692319    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-cni-bin\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692322    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce48bcb1-ccef-4183-af51-8a88c043cb64-serviceca\") pod \"node-ca-s9w8v\" (UID: \"ce48bcb1-ccef-4183-af51-8a88c043cb64\") " pod="openshift-image-registry/node-ca-s9w8v"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692330    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-kubelet\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692340    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-run-ovn-kubernetes\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692346    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52e351d7-6ba9-4265-bf15-4fcc3604a693-env-overrides\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692379    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-cni-bin\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692386    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-systemd-units\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692417    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-node-log\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692441    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce48bcb1-ccef-4183-af51-8a88c043cb64-host\") pod \"node-ca-s9w8v\" (UID: \"ce48bcb1-ccef-4183-af51-8a88c043cb64\") " pod="openshift-image-registry/node-ca-s9w8v"
Jul 31 23:46:07.692655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692481    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-systemd-units\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692496    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce48bcb1-ccef-4183-af51-8a88c043cb64-host\") pod \"node-ca-s9w8v\" (UID: \"ce48bcb1-ccef-4183-af51-8a88c043cb64\") " pod="openshift-image-registry/node-ca-s9w8v"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692504    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692483    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-node-log\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692535    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52e351d7-6ba9-4265-bf15-4fcc3604a693-ovnkube-script-lib\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692556    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692560    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qncgn\" (UniqueName: \"kubernetes.io/projected/d23c6534-149f-4a0b-baea-af7e105c571c-kube-api-access-qncgn\") pod \"iptables-alerter-c6n68\" (UID: \"d23c6534-149f-4a0b-baea-af7e105c571c\") " pod="openshift-network-operator/iptables-alerter-c6n68"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692596    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-etc-openvswitch\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692620    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-log-socket\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692645    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/776b1ef2-f435-44ca-9f0e-90316affeef8-konnectivity-ca\") pod \"konnectivity-agent-dhhpx\" (UID: \"776b1ef2-f435-44ca-9f0e-90316affeef8\") " pod="kube-system/konnectivity-agent-dhhpx"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692670    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692696    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-run-systemd\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692706    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-log-socket\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692720    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d23c6534-149f-4a0b-baea-af7e105c571c-iptables-alerter-script\") pod \"iptables-alerter-c6n68\" (UID: \"d23c6534-149f-4a0b-baea-af7e105c571c\") " pod="openshift-network-operator/iptables-alerter-c6n68"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692765    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52e351d7-6ba9-4265-bf15-4fcc3604a693-ovnkube-config\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692779    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-etc-openvswitch\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692791    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52e351d7-6ba9-4265-bf15-4fcc3604a693-env-overrides\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.693508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692796    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-var-lib-openvswitch\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.694073 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692826    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-var-lib-openvswitch\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.694073 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692845    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-run-openvswitch\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.694073 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:07.692868    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:07.694073 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.692883    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-run-openvswitch\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.694073 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:07.692926    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret podName:b717d75a-de98-4e7b-b86e-1af68be8f6a7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:08.192909724 +0000 UTC m=+3.066553320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret") pod "global-pull-secret-syncer-sjj4w" (UID: "b717d75a-de98-4e7b-b86e-1af68be8f6a7") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:07.694073 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.693158    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52e351d7-6ba9-4265-bf15-4fcc3604a693-run-systemd\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.694073 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.693189    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52e351d7-6ba9-4265-bf15-4fcc3604a693-ovnkube-script-lib\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.694073 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.693348    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d23c6534-149f-4a0b-baea-af7e105c571c-iptables-alerter-script\") pod \"iptables-alerter-c6n68\" (UID: \"d23c6534-149f-4a0b-baea-af7e105c571c\") " pod="openshift-network-operator/iptables-alerter-c6n68"
Jul 31 23:46:07.694073 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.693393    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52e351d7-6ba9-4265-bf15-4fcc3604a693-ovnkube-config\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.694073 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.693427    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/776b1ef2-f435-44ca-9f0e-90316affeef8-konnectivity-ca\") pod \"konnectivity-agent-dhhpx\" (UID: \"776b1ef2-f435-44ca-9f0e-90316affeef8\") " pod="kube-system/konnectivity-agent-dhhpx"
Jul 31 23:46:07.694457 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.694436    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52e351d7-6ba9-4265-bf15-4fcc3604a693-ovn-node-metrics-cert\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.694681 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.694659    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/776b1ef2-f435-44ca-9f0e-90316affeef8-agent-certs\") pod \"konnectivity-agent-dhhpx\" (UID: \"776b1ef2-f435-44ca-9f0e-90316affeef8\") " pod="kube-system/konnectivity-agent-dhhpx"
Jul 31 23:46:07.699345 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:07.699324    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:07.699438 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:07.699353    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:07.699438 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:07.699368    2584 projected.go:196] Error preparing data for projected volume kube-api-access-h2f6w for pod openshift-network-diagnostics/network-check-target-qpmwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:07.699438 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:07.699434    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w podName:6ca51531-7776-413a-98b3-aebadd923730 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:08.199416026 +0000 UTC m=+3.073059634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h2f6w" (UniqueName: "kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w") pod "network-check-target-qpmwv" (UID: "6ca51531-7776-413a-98b3-aebadd923730") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:07.701484 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.701465    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zg94\" (UniqueName: \"kubernetes.io/projected/52e351d7-6ba9-4265-bf15-4fcc3604a693-kube-api-access-9zg94\") pod \"ovnkube-node-khzpq\" (UID: \"52e351d7-6ba9-4265-bf15-4fcc3604a693\") " pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.701933 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.701914    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qncgn\" (UniqueName: \"kubernetes.io/projected/d23c6534-149f-4a0b-baea-af7e105c571c-kube-api-access-qncgn\") pod \"iptables-alerter-c6n68\" (UID: \"d23c6534-149f-4a0b-baea-af7e105c571c\") " pod="openshift-network-operator/iptables-alerter-c6n68"
Jul 31 23:46:07.702039 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.702026    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6vx7\" (UniqueName: \"kubernetes.io/projected/ce48bcb1-ccef-4183-af51-8a88c043cb64-kube-api-access-l6vx7\") pod \"node-ca-s9w8v\" (UID: \"ce48bcb1-ccef-4183-af51-8a88c043cb64\") " pod="openshift-image-registry/node-ca-s9w8v"
Jul 31 23:46:07.783098 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.783021    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w"
Jul 31 23:46:07.792063 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.792040    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q7xzv"
Jul 31 23:46:07.799796 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.799774    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hsjmv"
Jul 31 23:46:07.804596 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.804580    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-glccn"
Jul 31 23:46:07.811046 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.811029    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7hjxn"
Jul 31 23:46:07.817321 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.817305    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c6n68"
Jul 31 23:46:07.823859 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.823843    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s9w8v"
Jul 31 23:46:07.830559 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.830543    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:07.836057 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:07.836036    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dhhpx"
Jul 31 23:46:08.095376 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.095296    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs\") pod \"network-metrics-daemon-hsnnq\" (UID: \"9d3952c7-cede-4f37-aba3-9b96b84eaeb6\") " pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:08.095544 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:08.095406    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:08.095544 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:08.095469    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs podName:9d3952c7-cede-4f37-aba3-9b96b84eaeb6 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:09.095450642 +0000 UTC m=+3.969094235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs") pod "network-metrics-daemon-hsnnq" (UID: "9d3952c7-cede-4f37-aba3-9b96b84eaeb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:08.196403 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.196369    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:08.196586 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:08.196522    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:08.196647 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:08.196593    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret podName:b717d75a-de98-4e7b-b86e-1af68be8f6a7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:09.196579338 +0000 UTC m=+4.070222931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret") pod "global-pull-secret-syncer-sjj4w" (UID: "b717d75a-de98-4e7b-b86e-1af68be8f6a7") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:08.296766 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.296738    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2f6w\" (UniqueName: \"kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w\") pod \"network-check-target-qpmwv\" (UID: \"6ca51531-7776-413a-98b3-aebadd923730\") " pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:08.297007 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:08.296920    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:08.297007 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:08.296942    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:08.297007 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:08.296951    2584 projected.go:196] Error preparing data for projected volume kube-api-access-h2f6w for pod openshift-network-diagnostics/network-check-target-qpmwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:08.297175 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:08.297028    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w podName:6ca51531-7776-413a-98b3-aebadd923730 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:09.297011963 +0000 UTC m=+4.170655551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2f6w" (UniqueName: "kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w") pod "network-check-target-qpmwv" (UID: "6ca51531-7776-413a-98b3-aebadd923730") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:08.322147 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:08.322121    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3079ae51_76cf_4bde_90bf_05e1396256b3.slice/crio-36d385d0ce4a96bee0d244ae8f2400ea6b5a460823bae8400d1e6a3700c44ae1 WatchSource:0}: Error finding container 36d385d0ce4a96bee0d244ae8f2400ea6b5a460823bae8400d1e6a3700c44ae1: Status 404 returned error can't find the container with id 36d385d0ce4a96bee0d244ae8f2400ea6b5a460823bae8400d1e6a3700c44ae1
Jul 31 23:46:08.323306 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:08.323264    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3372e998_c5cb_4cd3_a17e_d8fcfec8a2fc.slice/crio-fe4653485e4cd47e55b911374ca54b17ac97d2f25d16ce8e6e7252222f708251 WatchSource:0}: Error finding container fe4653485e4cd47e55b911374ca54b17ac97d2f25d16ce8e6e7252222f708251: Status 404 returned error can't find the container with id fe4653485e4cd47e55b911374ca54b17ac97d2f25d16ce8e6e7252222f708251
Jul 31 23:46:08.325090 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:08.325061    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce48bcb1_ccef_4183_af51_8a88c043cb64.slice/crio-c2ec2b4cadd18eec8abc159741ec3b046c697973c49c70afdd2357694d6c0afd WatchSource:0}: Error finding container c2ec2b4cadd18eec8abc159741ec3b046c697973c49c70afdd2357694d6c0afd: Status 404 returned error can't find the container with id c2ec2b4cadd18eec8abc159741ec3b046c697973c49c70afdd2357694d6c0afd
Jul 31 23:46:08.328803 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:08.328370    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054662f6_b4fc_47bf_afbc_dee3eec7a1a7.slice/crio-b3a46cf6a87d2e1e521a2672a83e72eb26263015af1455623157b157586ecb33 WatchSource:0}: Error finding container b3a46cf6a87d2e1e521a2672a83e72eb26263015af1455623157b157586ecb33: Status 404 returned error can't find the container with id b3a46cf6a87d2e1e521a2672a83e72eb26263015af1455623157b157586ecb33
Jul 31 23:46:08.328803 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:08.328683    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df3a7f4_80f0_4a56_a857_d59531130023.slice/crio-5eac2ac39c644d45ce9f061ea3f88fb24eaf0536e20f91d3b50c6b0483f946f2 WatchSource:0}: Error finding container 5eac2ac39c644d45ce9f061ea3f88fb24eaf0536e20f91d3b50c6b0483f946f2: Status 404 returned error can't find the container with id 5eac2ac39c644d45ce9f061ea3f88fb24eaf0536e20f91d3b50c6b0483f946f2
Jul 31 23:46:08.329457 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:08.329430    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e351d7_6ba9_4265_bf15_4fcc3604a693.slice/crio-3482a4baba1cc073315eab4f985e05ecdaf1d1d5d8ef71788542aee36e73c485 WatchSource:0}: Error finding container 3482a4baba1cc073315eab4f985e05ecdaf1d1d5d8ef71788542aee36e73c485: Status 404 returned error can't find the container with id 3482a4baba1cc073315eab4f985e05ecdaf1d1d5d8ef71788542aee36e73c485
Jul 31 23:46:08.518471 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.518390    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 23:41:07 +0000 UTC" deadline="2028-04-26 13:31:54.413886544 +0000 UTC"
Jul 31 23:46:08.518471 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.518471    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15229h45m45.895425199s"
Jul 31 23:46:08.565449 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.565429    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:08.565544 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:08.565517    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:08.573873 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.573848    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glccn" event={"ID":"054662f6-b4fc-47bf-afbc-dee3eec7a1a7","Type":"ContainerStarted","Data":"b3a46cf6a87d2e1e521a2672a83e72eb26263015af1455623157b157586ecb33"}
Jul 31 23:46:08.574851 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.574831    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q7xzv" event={"ID":"3079ae51-76cf-4bde-90bf-05e1396256b3","Type":"ContainerStarted","Data":"36d385d0ce4a96bee0d244ae8f2400ea6b5a460823bae8400d1e6a3700c44ae1"}
Jul 31 23:46:08.576563 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.576535    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal" event={"ID":"4f0884df17fa2ea19d188a1590c26af3","Type":"ContainerStarted","Data":"ae16fbd2ae2ffc2f6e926857a840f52a1c7c40906595a674f89573d1078aeb0c"}
Jul 31 23:46:08.577509 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.577491    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w" event={"ID":"984d46fc-6501-4bbb-8b1f-c3bd36b84718","Type":"ContainerStarted","Data":"366014bcaff8fdc7ec415f3e42ecae0883a5ce8bb0f0f549641a227300f99275"}
Jul 31 23:46:08.578467 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.578449    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c6n68" event={"ID":"d23c6534-149f-4a0b-baea-af7e105c571c","Type":"ContainerStarted","Data":"912eedf0ea48e5c36c421cdcea11130eb86995e8ceb2785a2d171e35ad7b5c5d"}
Jul 31 23:46:08.579505 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.579482    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq" event={"ID":"52e351d7-6ba9-4265-bf15-4fcc3604a693","Type":"ContainerStarted","Data":"3482a4baba1cc073315eab4f985e05ecdaf1d1d5d8ef71788542aee36e73c485"}
Jul 31 23:46:08.580413 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.580393    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hsjmv" event={"ID":"1df3a7f4-80f0-4a56-a857-d59531130023","Type":"ContainerStarted","Data":"5eac2ac39c644d45ce9f061ea3f88fb24eaf0536e20f91d3b50c6b0483f946f2"}
Jul 31 23:46:08.581362 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.581344    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s9w8v" event={"ID":"ce48bcb1-ccef-4183-af51-8a88c043cb64","Type":"ContainerStarted","Data":"c2ec2b4cadd18eec8abc159741ec3b046c697973c49c70afdd2357694d6c0afd"}
Jul 31 23:46:08.582421 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.582398    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7hjxn" event={"ID":"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc","Type":"ContainerStarted","Data":"fe4653485e4cd47e55b911374ca54b17ac97d2f25d16ce8e6e7252222f708251"}
Jul 31 23:46:08.583675 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.583657    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-252.ec2.internal" event={"ID":"e413b343bfdc4c93b43b646e37497322","Type":"ContainerStarted","Data":"9b081323b2255bc23b8279c3276971dde551370cab658bc17c690c511c03bf52"}
Jul 31 23:46:08.584671 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:08.584653    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dhhpx" event={"ID":"776b1ef2-f435-44ca-9f0e-90316affeef8","Type":"ContainerStarted","Data":"4b55f7e15e17dc43f30247ede2d06d813ed44c6ca8ea8385bd0c8b1c6c5b5f9a"}
Jul 31 23:46:09.105203 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:09.105079    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs\") pod \"network-metrics-daemon-hsnnq\" (UID: \"9d3952c7-cede-4f37-aba3-9b96b84eaeb6\") " pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:09.105378 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:09.105270    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:09.105378 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:09.105334    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs podName:9d3952c7-cede-4f37-aba3-9b96b84eaeb6 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:11.105316305 +0000 UTC m=+5.978959909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs") pod "network-metrics-daemon-hsnnq" (UID: "9d3952c7-cede-4f37-aba3-9b96b84eaeb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:09.205529 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:09.205490    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:09.205712 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:09.205647    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:09.205771 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:09.205718    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret podName:b717d75a-de98-4e7b-b86e-1af68be8f6a7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:11.205699255 +0000 UTC m=+6.079342845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret") pod "global-pull-secret-syncer-sjj4w" (UID: "b717d75a-de98-4e7b-b86e-1af68be8f6a7") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:09.306186 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:09.306126    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2f6w\" (UniqueName: \"kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w\") pod \"network-check-target-qpmwv\" (UID: \"6ca51531-7776-413a-98b3-aebadd923730\") " pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:09.306373 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:09.306350    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:09.306373 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:09.306371    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:09.306487 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:09.306384    2584 projected.go:196] Error preparing data for projected volume kube-api-access-h2f6w for pod openshift-network-diagnostics/network-check-target-qpmwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:09.306487 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:09.306464    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w podName:6ca51531-7776-413a-98b3-aebadd923730 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:11.306424783 +0000 UTC m=+6.180068389 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2f6w" (UniqueName: "kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w") pod "network-check-target-qpmwv" (UID: "6ca51531-7776-413a-98b3-aebadd923730") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:09.519285 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:09.519190    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 23:41:07 +0000 UTC" deadline="2028-04-07 00:36:49.130068133 +0000 UTC"
Jul 31 23:46:09.519285 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:09.519231    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14760h50m39.610844153s"
Jul 31 23:46:09.580451 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:09.580159    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:09.580451 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:09.580296    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:09.580451 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:09.580337    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:09.580451 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:09.580419    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:09.606163 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:09.606117    2584 generic.go:356] "Generic (PLEG): container finished" podID="4f0884df17fa2ea19d188a1590c26af3" containerID="ae16fbd2ae2ffc2f6e926857a840f52a1c7c40906595a674f89573d1078aeb0c" exitCode=0
Jul 31 23:46:09.606716 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:09.606693    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal" event={"ID":"4f0884df17fa2ea19d188a1590c26af3","Type":"ContainerDied","Data":"ae16fbd2ae2ffc2f6e926857a840f52a1c7c40906595a674f89573d1078aeb0c"}
Jul 31 23:46:09.626299 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:09.626240    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-252.ec2.internal" podStartSLOduration=3.626219658 podStartE2EDuration="3.626219658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:08.605354931 +0000 UTC m=+3.478998540" watchObservedRunningTime="2026-07-31 23:46:09.626219658 +0000 UTC m=+4.499863269"
Jul 31 23:46:10.566126 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:10.566095    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:10.566579 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:10.566228    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:10.627516 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:10.627473    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal" event={"ID":"4f0884df17fa2ea19d188a1590c26af3","Type":"ContainerStarted","Data":"1a720bf3e8ca7e650d4746067d74b116404259a2b70ffc4de3c6f7eb8cd55993"}
Jul 31 23:46:10.654753 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:10.652581    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-252.ec2.internal" podStartSLOduration=4.65255563 podStartE2EDuration="4.65255563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:10.65120824 +0000 UTC m=+5.524851852" watchObservedRunningTime="2026-07-31 23:46:10.65255563 +0000 UTC m=+5.526199241"
Jul 31 23:46:11.120708 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:11.120671    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs\") pod \"network-metrics-daemon-hsnnq\" (UID: \"9d3952c7-cede-4f37-aba3-9b96b84eaeb6\") " pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:11.120891 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:11.120847    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:11.120953 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:11.120908    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs podName:9d3952c7-cede-4f37-aba3-9b96b84eaeb6 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:15.120889547 +0000 UTC m=+9.994533150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs") pod "network-metrics-daemon-hsnnq" (UID: "9d3952c7-cede-4f37-aba3-9b96b84eaeb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:11.221204 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:11.221128    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:11.221391 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:11.221305    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:11.221391 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:11.221372    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret podName:b717d75a-de98-4e7b-b86e-1af68be8f6a7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:15.221353717 +0000 UTC m=+10.094997308 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret") pod "global-pull-secret-syncer-sjj4w" (UID: "b717d75a-de98-4e7b-b86e-1af68be8f6a7") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:11.322650 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:11.321914    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2f6w\" (UniqueName: \"kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w\") pod \"network-check-target-qpmwv\" (UID: \"6ca51531-7776-413a-98b3-aebadd923730\") " pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:11.322650 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:11.322178    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:11.322650 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:11.322200    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:11.322650 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:11.322212    2584 projected.go:196] Error preparing data for projected volume kube-api-access-h2f6w for pod openshift-network-diagnostics/network-check-target-qpmwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:11.322650 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:11.322271    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w podName:6ca51531-7776-413a-98b3-aebadd923730 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:15.322252387 +0000 UTC m=+10.195896007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2f6w" (UniqueName: "kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w") pod "network-check-target-qpmwv" (UID: "6ca51531-7776-413a-98b3-aebadd923730") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:11.565819 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:11.565354    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:11.565819 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:11.565487    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:11.565819 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:11.565550    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:11.565819 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:11.565752    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:12.565230 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:12.565201    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:12.565429 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:12.565328    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:13.567867 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:13.567459    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:13.567867 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:13.567602    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:13.567867 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:13.567652    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:13.567867 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:13.567779    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:14.565310 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:14.565271    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:14.565500 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:14.565419    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:15.157950 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:15.157910    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs\") pod \"network-metrics-daemon-hsnnq\" (UID: \"9d3952c7-cede-4f37-aba3-9b96b84eaeb6\") " pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:15.158428 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:15.158076    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:15.158428 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:15.158164    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs podName:9d3952c7-cede-4f37-aba3-9b96b84eaeb6 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:23.158141755 +0000 UTC m=+18.031785346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs") pod "network-metrics-daemon-hsnnq" (UID: "9d3952c7-cede-4f37-aba3-9b96b84eaeb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:15.258771 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:15.258737    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:15.258935 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:15.258874    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:15.259013 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:15.258936    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret podName:b717d75a-de98-4e7b-b86e-1af68be8f6a7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:23.258918401 +0000 UTC m=+18.132562005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret") pod "global-pull-secret-syncer-sjj4w" (UID: "b717d75a-de98-4e7b-b86e-1af68be8f6a7") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:15.359906 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:15.359802    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2f6w\" (UniqueName: \"kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w\") pod \"network-check-target-qpmwv\" (UID: \"6ca51531-7776-413a-98b3-aebadd923730\") " pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:15.360095 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:15.359985    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:15.360095 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:15.360009    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:15.360095 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:15.360022    2584 projected.go:196] Error preparing data for projected volume kube-api-access-h2f6w for pod openshift-network-diagnostics/network-check-target-qpmwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:15.360095 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:15.360073    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w podName:6ca51531-7776-413a-98b3-aebadd923730 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:23.360056561 +0000 UTC m=+18.233700154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2f6w" (UniqueName: "kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w") pod "network-check-target-qpmwv" (UID: "6ca51531-7776-413a-98b3-aebadd923730") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:15.567034 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:15.566932    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:15.567197 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:15.567076    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:15.567197 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:15.567111    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:15.567302 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:15.567223    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:16.565369 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:16.565295    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:16.565744 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:16.565441    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:17.565536 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:17.565500    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:17.565951 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:17.565621    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:17.565951 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:17.565718    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:17.565951 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:17.565845    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:18.565323 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:18.565281    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:18.565496 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:18.565396    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:19.565330 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:19.565297    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:19.565330 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:19.565317    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:19.565850 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:19.565430    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:19.565850 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:19.565560    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:20.566004 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:20.565954    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:20.566428 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:20.566107    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:21.565177 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:21.565141    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:21.565351 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:21.565141    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:21.565351 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:21.565291    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:21.565351 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:21.565342    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:22.565875 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:22.565847    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:22.566363 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:22.565975    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:23.222484 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:23.222446    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs\") pod \"network-metrics-daemon-hsnnq\" (UID: \"9d3952c7-cede-4f37-aba3-9b96b84eaeb6\") " pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:23.222655 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:23.222596    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:23.222702 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:23.222667    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs podName:9d3952c7-cede-4f37-aba3-9b96b84eaeb6 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:39.222647115 +0000 UTC m=+34.096290707 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs") pod "network-metrics-daemon-hsnnq" (UID: "9d3952c7-cede-4f37-aba3-9b96b84eaeb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:23.323498 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:23.323461    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:23.323680 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:23.323569    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:23.323680 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:23.323628    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret podName:b717d75a-de98-4e7b-b86e-1af68be8f6a7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:39.32360823 +0000 UTC m=+34.197251818 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret") pod "global-pull-secret-syncer-sjj4w" (UID: "b717d75a-de98-4e7b-b86e-1af68be8f6a7") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:23.424675 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:23.424597    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:23.424675 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:23.424642    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:23.424675 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:23.424656    2584 projected.go:196] Error preparing data for projected volume kube-api-access-h2f6w for pod openshift-network-diagnostics/network-check-target-qpmwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:23.424886 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:23.424722    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w podName:6ca51531-7776-413a-98b3-aebadd923730 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:39.42469835 +0000 UTC m=+34.298341939 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2f6w" (UniqueName: "kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w") pod "network-check-target-qpmwv" (UID: "6ca51531-7776-413a-98b3-aebadd923730") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:23.425208 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:23.424468    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2f6w\" (UniqueName: \"kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w\") pod \"network-check-target-qpmwv\" (UID: \"6ca51531-7776-413a-98b3-aebadd923730\") " pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:23.565916 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:23.565876    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:23.566328 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:23.565877    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:23.566328 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:23.566045    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:23.566328 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:23.566094    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:24.565844 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:24.565811    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:24.566057 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:24.565922    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:25.565943 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:25.565914    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:25.566125 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:25.566028    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:25.566125 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:25.566105    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:25.566564 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:25.566219    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:26.565506 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:26.565471    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:26.565692 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:26.565584    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:27.566086 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.565647    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:27.566685 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.565653    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:27.566685 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:27.566210    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:27.566685 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:27.566241    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:27.662177 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.662149    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dhhpx" event={"ID":"776b1ef2-f435-44ca-9f0e-90316affeef8","Type":"ContainerStarted","Data":"94f6c2e91b23913f303bbba4152de48bc1a3fa0a94eff57499fc6fa81f5d884d"}
Jul 31 23:46:27.663946 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.663919    2584 generic.go:356] "Generic (PLEG): container finished" podID="054662f6-b4fc-47bf-afbc-dee3eec7a1a7" containerID="c0347ea5bf53f13a90fb6b6b08c7d7589130fcef291234011fde16636dcdc53a" exitCode=0
Jul 31 23:46:27.664051 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.663998    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glccn" event={"ID":"054662f6-b4fc-47bf-afbc-dee3eec7a1a7","Type":"ContainerDied","Data":"c0347ea5bf53f13a90fb6b6b08c7d7589130fcef291234011fde16636dcdc53a"}
Jul 31 23:46:27.666250 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.666222    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q7xzv" event={"ID":"3079ae51-76cf-4bde-90bf-05e1396256b3","Type":"ContainerStarted","Data":"00000528731fd455a01b61f351820c6ab51d12a98c91d6f7613dca844e481603"}
Jul 31 23:46:27.667698 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.667680    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w" event={"ID":"984d46fc-6501-4bbb-8b1f-c3bd36b84718","Type":"ContainerStarted","Data":"768f8bcb7854db1c001f19c7b86e544fd8fcdb581e4adce6253fbbc04adbe58d"}
Jul 31 23:46:27.670515 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.670495    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq" event={"ID":"52e351d7-6ba9-4265-bf15-4fcc3604a693","Type":"ContainerStarted","Data":"097d69a66abc75d1982cb3fca62bcc5a9bba6a1b78b955ebb058bae579a88995"}
Jul 31 23:46:27.670632 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.670548    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq" event={"ID":"52e351d7-6ba9-4265-bf15-4fcc3604a693","Type":"ContainerStarted","Data":"10ebdd35a9f64d7fbe90756ec7120e0cbbefefb013a9ce0b4419bdb0225140cb"}
Jul 31 23:46:27.670632 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.670590    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq" event={"ID":"52e351d7-6ba9-4265-bf15-4fcc3604a693","Type":"ContainerStarted","Data":"8f754cd26fd6fc6ce719a0177ab0cfee2a9153282838576420ff8fc74dae3299"}
Jul 31 23:46:27.670632 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.670619    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq" event={"ID":"52e351d7-6ba9-4265-bf15-4fcc3604a693","Type":"ContainerStarted","Data":"18042a902f85d6e48ec405fe78d0e4c8fa97457fee7d00f5306f642fc073fc79"}
Jul 31 23:46:27.670748 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.670651    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq" event={"ID":"52e351d7-6ba9-4265-bf15-4fcc3604a693","Type":"ContainerStarted","Data":"7fa0b2167f804b0875bffd9b3acd2a88a5a4c0905568d85e207c506c0d6121e6"}
Jul 31 23:46:27.671767 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.671751    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hsjmv" event={"ID":"1df3a7f4-80f0-4a56-a857-d59531130023","Type":"ContainerStarted","Data":"a33636fffa3fd67d4750117838dbcbb329168a46a46ccb2dbde158e91e7edb01"}
Jul 31 23:46:27.673257 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.673238    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s9w8v" event={"ID":"ce48bcb1-ccef-4183-af51-8a88c043cb64","Type":"ContainerStarted","Data":"3e7b632e100127d7936cc924b34206b86d1ed55ccee49a3de1f1e6d07e661176"}
Jul 31 23:46:27.674541 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.674525    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7hjxn" event={"ID":"3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc","Type":"ContainerStarted","Data":"16c4c827c939aad63c5bff057819f10e068e235a69c94bd4c6aaf9ae0c6c4ce4"}
Jul 31 23:46:27.694725 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.694643    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dhhpx" podStartSLOduration=7.926396543 podStartE2EDuration="22.694628771s" totalImagesPullingTime="14.768232228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:27.679058059 +0000 UTC m=+22.552701665" watchObservedRunningTime="2026-07-31 23:46:27.694628771 +0000 UTC m=+22.568272402"
Jul 31 23:46:27.695205 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.695175    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7hjxn" podStartSLOduration=4.019873802 podStartE2EDuration="22.695167238s" totalImagesPullingTime="18.675293436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:27.694528907 +0000 UTC m=+22.568172527" watchObservedRunningTime="2026-07-31 23:46:27.695167238 +0000 UTC m=+22.568810847"
Jul 31 23:46:27.710228 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.710169    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q7xzv" podStartSLOduration=4.074014824 podStartE2EDuration="22.71014907s" totalImagesPullingTime="18.636134246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:27.709942753 +0000 UTC m=+22.583586364" watchObservedRunningTime="2026-07-31 23:46:27.71014907 +0000 UTC m=+22.583792681"
Jul 31 23:46:27.726143 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.726092    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hsjmv" podStartSLOduration=4.124360785 podStartE2EDuration="22.726074106s" totalImagesPullingTime="18.601713321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:27.725336133 +0000 UTC m=+22.598979746" watchObservedRunningTime="2026-07-31 23:46:27.726074106 +0000 UTC m=+22.599717718"
Jul 31 23:46:27.777916 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:27.777859    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s9w8v" podStartSLOduration=4.171702083 podStartE2EDuration="22.777839216s" totalImagesPullingTime="18.606137133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:27.777389453 +0000 UTC m=+22.651033062" watchObservedRunningTime="2026-07-31 23:46:27.777839216 +0000 UTC m=+22.651482828"
Jul 31 23:46:28.454792 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:28.454763    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 23:46:28.551440 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:28.551318    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T23:46:28.454788054Z","UUID":"c9bf05e4-5e3e-4aad-81f3-ba5751b989f4","Handler":null,"Name":"","Endpoint":""}
Jul 31 23:46:28.552914 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:28.552886    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 23:46:28.553067 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:28.552941    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 23:46:28.566109 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:28.566080    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:28.566586 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:28.566200    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:28.679184 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:28.679096    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w" event={"ID":"984d46fc-6501-4bbb-8b1f-c3bd36b84718","Type":"ContainerStarted","Data":"8884f71c75b5db268c100a644c2ac7c332e62047a84b02c6e29547079fb2ca47"}
Jul 31 23:46:28.680740 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:28.680714    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c6n68" event={"ID":"d23c6534-149f-4a0b-baea-af7e105c571c","Type":"ContainerStarted","Data":"352b93154d2555a0a73e4d1634c12ce89f1e2993d57104cafa32ab6c98a2b2c2"}
Jul 31 23:46:28.684270 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:28.684244    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq" event={"ID":"52e351d7-6ba9-4265-bf15-4fcc3604a693","Type":"ContainerStarted","Data":"c695c8a60e61cf6cd9b10dc7cc5e88e57a954df505d02f2e997ca040ae30ad9e"}
Jul 31 23:46:28.696168 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:28.696107    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c6n68" podStartSLOduration=5.090953746 podStartE2EDuration="23.696085645s" totalImagesPullingTime="18.605131899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:28.695090524 +0000 UTC m=+23.568734135" watchObservedRunningTime="2026-07-31 23:46:28.696085645 +0000 UTC m=+23.569729254"
Jul 31 23:46:29.570866 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:29.570838    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:29.571394 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:29.570843    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:29.571394 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:29.571060    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:29.571394 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:29.570935    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:30.566076 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:30.566044    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:30.566256 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:30.566185    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:30.695380 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:30.695344    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w" event={"ID":"984d46fc-6501-4bbb-8b1f-c3bd36b84718","Type":"ContainerStarted","Data":"cd93d85a55340df6c75c495ee6d11cd0222742f7c269ebcede827a48264cf6c6"}
Jul 31 23:46:30.698910 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:30.698873    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq" event={"ID":"52e351d7-6ba9-4265-bf15-4fcc3604a693","Type":"ContainerStarted","Data":"1bf7874319e85d77a6ac9fe50e47abefc47a0891c46b0c95bd7153e73182c48a"}
Jul 31 23:46:30.712882 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:30.712836    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85q4w" podStartSLOduration=4.80885727 podStartE2EDuration="25.712819048s" totalImagesPullingTime="20.903961778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:30.712389362 +0000 UTC m=+25.586032985" watchObservedRunningTime="2026-07-31 23:46:30.712819048 +0000 UTC m=+25.586462660"
Jul 31 23:46:31.565906 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:31.565872    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:31.566110 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:31.565871    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:31.566110 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:31.566032    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:31.566110 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:31.566091    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:32.199713 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:32.199686    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dhhpx"
Jul 31 23:46:32.200505 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:32.200486    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dhhpx"
Jul 31 23:46:32.565713 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:32.565463    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:32.565848 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:32.565722    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:32.707568 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:32.707528    2584 generic.go:356] "Generic (PLEG): container finished" podID="054662f6-b4fc-47bf-afbc-dee3eec7a1a7" containerID="8d63568a0b421cb998a4f40c8293246936a02d84e29b1d4a4720f4377460c7c8" exitCode=0
Jul 31 23:46:32.707748 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:32.707622    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glccn" event={"ID":"054662f6-b4fc-47bf-afbc-dee3eec7a1a7","Type":"ContainerDied","Data":"8d63568a0b421cb998a4f40c8293246936a02d84e29b1d4a4720f4377460c7c8"}
Jul 31 23:46:32.711255 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:32.711165    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq" event={"ID":"52e351d7-6ba9-4265-bf15-4fcc3604a693","Type":"ContainerStarted","Data":"142392f8bd3a82d1922a3c2adb55041410a5e326d6977b7dc838d92e8975837a"}
Jul 31 23:46:32.711471 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:32.711451    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dhhpx"
Jul 31 23:46:32.711864 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:32.711841    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dhhpx"
Jul 31 23:46:32.799778 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:32.799723    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq" podStartSLOduration=9.127932562 podStartE2EDuration="27.799681105s" totalImagesPullingTime="18.671748543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:32.778702595 +0000 UTC m=+27.652346206" watchObservedRunningTime="2026-07-31 23:46:32.799681105 +0000 UTC m=+27.673324716"
Jul 31 23:46:33.565557 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:33.565520    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:33.566197 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:33.565640    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:33.566197 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:33.566107    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:33.566325 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:33.566216    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:33.713598 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:33.713532    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:33.713742 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:33.713607    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:33.713742 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:33.713649    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:33.728082 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:33.728057    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:33.728249 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:33.728238    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:46:34.269109 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:34.269030    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qpmwv"]
Jul 31 23:46:34.269275 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:34.269212    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:34.269335 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:34.269299    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:34.273250 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:34.273226    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sjj4w"]
Jul 31 23:46:34.273361 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:34.273326    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:34.273424 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:34.273408    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:34.283448 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:34.283422    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hsnnq"]
Jul 31 23:46:34.283600 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:34.283526    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:34.283654 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:34.283627    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:34.717451 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:34.717363    2584 generic.go:356] "Generic (PLEG): container finished" podID="054662f6-b4fc-47bf-afbc-dee3eec7a1a7" containerID="c491b633842358a437d033b68da32ff3c80958b630b9a3449f0eb18b40e1a7eb" exitCode=0
Jul 31 23:46:34.717910 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:34.717449    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glccn" event={"ID":"054662f6-b4fc-47bf-afbc-dee3eec7a1a7","Type":"ContainerDied","Data":"c491b633842358a437d033b68da32ff3c80958b630b9a3449f0eb18b40e1a7eb"}
Jul 31 23:46:35.566495 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:35.566465    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:35.566694 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:35.566552    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:35.566694 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:35.566583    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:35.566694 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:35.566620    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:35.566694 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:35.566643    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:35.566892 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:35.566700    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:35.725330 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:35.725300    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glccn" event={"ID":"054662f6-b4fc-47bf-afbc-dee3eec7a1a7","Type":"ContainerStarted","Data":"662a7a2db7c616f17ca287381821d117a17517730ac44e5592e51678f486517c"}
Jul 31 23:46:36.729804 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:36.729496    2584 generic.go:356] "Generic (PLEG): container finished" podID="054662f6-b4fc-47bf-afbc-dee3eec7a1a7" containerID="662a7a2db7c616f17ca287381821d117a17517730ac44e5592e51678f486517c" exitCode=0
Jul 31 23:46:36.729804 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:36.729577    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glccn" event={"ID":"054662f6-b4fc-47bf-afbc-dee3eec7a1a7","Type":"ContainerDied","Data":"662a7a2db7c616f17ca287381821d117a17517730ac44e5592e51678f486517c"}
Jul 31 23:46:37.565286 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:37.565256    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:37.565450 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:37.565294    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:37.565450 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:37.565257    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:37.565450 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:37.565423    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:37.565604 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:37.565517    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:37.565647 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:37.565625    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:39.249016 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.248974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs\") pod \"network-metrics-daemon-hsnnq\" (UID: \"9d3952c7-cede-4f37-aba3-9b96b84eaeb6\") " pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:39.249485 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:39.249087    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:39.249485 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:39.249148    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs podName:9d3952c7-cede-4f37-aba3-9b96b84eaeb6 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:11.24913105 +0000 UTC m=+66.122774644 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs") pod "network-metrics-daemon-hsnnq" (UID: "9d3952c7-cede-4f37-aba3-9b96b84eaeb6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:39.350161 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.350127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:39.350328 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:39.350260    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:39.350328 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:39.350318    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret podName:b717d75a-de98-4e7b-b86e-1af68be8f6a7 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:11.350304185 +0000 UTC m=+66.223947789 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret") pod "global-pull-secret-syncer-sjj4w" (UID: "b717d75a-de98-4e7b-b86e-1af68be8f6a7") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:39.451409 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.451380    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2f6w\" (UniqueName: \"kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w\") pod \"network-check-target-qpmwv\" (UID: \"6ca51531-7776-413a-98b3-aebadd923730\") " pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:39.451547 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:39.451524    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:39.451547 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:39.451540    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:39.451547 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:39.451548    2584 projected.go:196] Error preparing data for projected volume kube-api-access-h2f6w for pod openshift-network-diagnostics/network-check-target-qpmwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:39.451773 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:39.451600    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w podName:6ca51531-7776-413a-98b3-aebadd923730 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:11.451581249 +0000 UTC m=+66.325224853 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2f6w" (UniqueName: "kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w") pod "network-check-target-qpmwv" (UID: "6ca51531-7776-413a-98b3-aebadd923730") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:39.565669 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.565596    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:39.565669 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.565648    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:39.565922 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.565648    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:39.565922 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:39.565735    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qpmwv" podUID="6ca51531-7776-413a-98b3-aebadd923730"
Jul 31 23:46:39.565922 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:39.565832    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsnnq" podUID="9d3952c7-cede-4f37-aba3-9b96b84eaeb6"
Jul 31 23:46:39.565922 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:39.565898    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjj4w" podUID="b717d75a-de98-4e7b-b86e-1af68be8f6a7"
Jul 31 23:46:39.936414 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.936386    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-252.ec2.internal" event="NodeReady"
Jul 31 23:46:39.936573 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.936538    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 23:46:39.973550 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.973524    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"]
Jul 31 23:46:39.976290 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.976272    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:39.978892 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.978873    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 31 23:46:39.978994 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.978912    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 31 23:46:39.979054 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.979017    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 31 23:46:39.979125 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.979106    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mbvf9\""
Jul 31 23:46:39.984770 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.984533    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 31 23:46:39.993349 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.993326    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zmrgg"]
Jul 31 23:46:39.995316 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.995296    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:39.998172 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.997987    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 23:46:39.998656 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.998636    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r2mdc\""
Jul 31 23:46:39.999471 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:39.999449    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 23:46:40.000536 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.000506    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ghqfz"]
Jul 31 23:46:40.002326 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.002112    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"]
Jul 31 23:46:40.002326 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.002218    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ghqfz"
Jul 31 23:46:40.005161 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.005022    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 23:46:40.005161 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.005048    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 23:46:40.005310 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.005294    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 23:46:40.005400 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.005379    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-btr74\""
Jul 31 23:46:40.007762 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.007741    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zmrgg"]
Jul 31 23:46:40.010603 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.010583    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ghqfz"]
Jul 31 23:46:40.056561 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.056527    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ea4e9da-91c8-4410-969d-49f53895860e-registry-certificates\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.056721 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.056569    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ea4e9da-91c8-4410-969d-49f53895860e-trusted-ca\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.056721 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.056596    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ea4e9da-91c8-4410-969d-49f53895860e-ca-trust-extracted\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.056721 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.056652    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-bound-sa-token\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.056721 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.056712    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ea4e9da-91c8-4410-969d-49f53895860e-installation-pull-secrets\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.056919 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.056760    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.056919 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.056842    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ea4e9da-91c8-4410-969d-49f53895860e-image-registry-private-configuration\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.056919 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.056877    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-572ss\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-kube-api-access-572ss\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.157868 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.157833    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ea4e9da-91c8-4410-969d-49f53895860e-trusted-ca\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.157868 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.157874    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ea4e9da-91c8-4410-969d-49f53895860e-ca-trust-extracted\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.158141 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.157895    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-bound-sa-token\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.158141 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.157922    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-tmp-dir\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:40.158141 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.157949    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ea4e9da-91c8-4410-969d-49f53895860e-installation-pull-secrets\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.158141 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.157990    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:40.158141 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.158034    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.158141 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.158057    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af99387b-4f11-481c-a86d-aa666c795d94-cert\") pod \"ingress-canary-ghqfz\" (UID: \"af99387b-4f11-481c-a86d-aa666c795d94\") " pod="openshift-ingress-canary/ingress-canary-ghqfz"
Jul 31 23:46:40.158141 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.158100    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ea4e9da-91c8-4410-969d-49f53895860e-image-registry-private-configuration\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.158489 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:40.158155    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:40.158489 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:40.158177    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x: secret "image-registry-tls" not found
Jul 31 23:46:40.158489 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:40.158232    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls podName:3ea4e9da-91c8-4410-969d-49f53895860e nodeName:}" failed. No retries permitted until 2026-07-31 23:46:40.65821312 +0000 UTC m=+35.531856722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls") pod "image-registry-5cfbfcfdcd-tlc8x" (UID: "3ea4e9da-91c8-4410-969d-49f53895860e") : secret "image-registry-tls" not found
Jul 31 23:46:40.158489 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.158157    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-572ss\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-kube-api-access-572ss\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.158489 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.158334    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w8pb\" (UniqueName: \"kubernetes.io/projected/af99387b-4f11-481c-a86d-aa666c795d94-kube-api-access-9w8pb\") pod \"ingress-canary-ghqfz\" (UID: \"af99387b-4f11-481c-a86d-aa666c795d94\") " pod="openshift-ingress-canary/ingress-canary-ghqfz"
Jul 31 23:46:40.158489 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.158370    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttbxh\" (UniqueName: \"kubernetes.io/projected/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-kube-api-access-ttbxh\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:40.158489 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.158392    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ea4e9da-91c8-4410-969d-49f53895860e-ca-trust-extracted\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.158489 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.158417    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ea4e9da-91c8-4410-969d-49f53895860e-registry-certificates\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.158489 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.158480    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-config-volume\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:40.159172 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.159145    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ea4e9da-91c8-4410-969d-49f53895860e-registry-certificates\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.159404 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.159385    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ea4e9da-91c8-4410-969d-49f53895860e-trusted-ca\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.163200 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.163173    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ea4e9da-91c8-4410-969d-49f53895860e-image-registry-private-configuration\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.163299 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.163231    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ea4e9da-91c8-4410-969d-49f53895860e-installation-pull-secrets\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.167037 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.167001    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-bound-sa-token\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.167294 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.167270    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-572ss\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-kube-api-access-572ss\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.259015 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.258916    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-tmp-dir\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:40.259015 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.258974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:40.259756 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:40.259134    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:40.259756 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.259193    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af99387b-4f11-481c-a86d-aa666c795d94-cert\") pod \"ingress-canary-ghqfz\" (UID: \"af99387b-4f11-481c-a86d-aa666c795d94\") " pod="openshift-ingress-canary/ingress-canary-ghqfz"
Jul 31 23:46:40.259756 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:40.259214    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls podName:f6ce1e39-8710-41b1-8fe8-c6ba227ad006 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:40.759192685 +0000 UTC m=+35.632836293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls") pod "dns-default-zmrgg" (UID: "f6ce1e39-8710-41b1-8fe8-c6ba227ad006") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:40.259756 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.259270    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w8pb\" (UniqueName: \"kubernetes.io/projected/af99387b-4f11-481c-a86d-aa666c795d94-kube-api-access-9w8pb\") pod \"ingress-canary-ghqfz\" (UID: \"af99387b-4f11-481c-a86d-aa666c795d94\") " pod="openshift-ingress-canary/ingress-canary-ghqfz"
Jul 31 23:46:40.259756 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.259298    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttbxh\" (UniqueName: \"kubernetes.io/projected/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-kube-api-access-ttbxh\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:40.259756 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.259329    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-tmp-dir\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:40.259756 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.259338    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-config-volume\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:40.260040 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.259871    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-config-volume\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:40.261866 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.261844    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af99387b-4f11-481c-a86d-aa666c795d94-cert\") pod \"ingress-canary-ghqfz\" (UID: \"af99387b-4f11-481c-a86d-aa666c795d94\") " pod="openshift-ingress-canary/ingress-canary-ghqfz"
Jul 31 23:46:40.268305 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.268285    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w8pb\" (UniqueName: \"kubernetes.io/projected/af99387b-4f11-481c-a86d-aa666c795d94-kube-api-access-9w8pb\") pod \"ingress-canary-ghqfz\" (UID: \"af99387b-4f11-481c-a86d-aa666c795d94\") " pod="openshift-ingress-canary/ingress-canary-ghqfz"
Jul 31 23:46:40.268838 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.268815    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttbxh\" (UniqueName: \"kubernetes.io/projected/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-kube-api-access-ttbxh\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:40.316040 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.316006    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ghqfz"
Jul 31 23:46:40.662666 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.662623    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:40.662845 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:40.662823    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:40.662920 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:40.662848    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x: secret "image-registry-tls" not found
Jul 31 23:46:40.662920 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:40.662909    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls podName:3ea4e9da-91c8-4410-969d-49f53895860e nodeName:}" failed. No retries permitted until 2026-07-31 23:46:41.662888728 +0000 UTC m=+36.536532318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls") pod "image-registry-5cfbfcfdcd-tlc8x" (UID: "3ea4e9da-91c8-4410-969d-49f53895860e") : secret "image-registry-tls" not found
Jul 31 23:46:40.710661 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.710509    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ghqfz"]
Jul 31 23:46:40.713112 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:40.713088    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf99387b_4f11_481c_a86d_aa666c795d94.slice/crio-a153ae6301805d9c4d3010c18dcf8d4d0894199dbc5136467a99315ae5826439 WatchSource:0}: Error finding container a153ae6301805d9c4d3010c18dcf8d4d0894199dbc5136467a99315ae5826439: Status 404 returned error can't find the container with id a153ae6301805d9c4d3010c18dcf8d4d0894199dbc5136467a99315ae5826439
Jul 31 23:46:40.742333 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.742291    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glccn" event={"ID":"054662f6-b4fc-47bf-afbc-dee3eec7a1a7","Type":"ContainerStarted","Data":"dfe209d9c028da0fede948e23668770c1c90acbb7ec9dd4ff0c70202eb331f0a"}
Jul 31 23:46:40.743443 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.743411    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ghqfz" event={"ID":"af99387b-4f11-481c-a86d-aa666c795d94","Type":"ContainerStarted","Data":"a153ae6301805d9c4d3010c18dcf8d4d0894199dbc5136467a99315ae5826439"}
Jul 31 23:46:40.763398 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:40.763356    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:40.763550 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:40.763510    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:40.763589 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:40.763573    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls podName:f6ce1e39-8710-41b1-8fe8-c6ba227ad006 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:41.763555083 +0000 UTC m=+36.637198672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls") pod "dns-default-zmrgg" (UID: "f6ce1e39-8710-41b1-8fe8-c6ba227ad006") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:41.565639 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:41.565602    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:46:41.566251 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:41.565602    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:46:41.566251 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:41.565606    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:46:41.568548 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:41.568522    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zhtdr\""
Jul 31 23:46:41.568688 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:41.568560    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 23:46:41.569592 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:41.569567    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n5wdr\""
Jul 31 23:46:41.569692 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:41.569641    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 23:46:41.569750 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:41.569587    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 23:46:41.569878 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:41.569852    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 23:46:41.669649 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:41.669614    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:41.669821 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:41.669758    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:41.669821 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:41.669775    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x: secret "image-registry-tls" not found
Jul 31 23:46:41.669932 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:41.669840    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls podName:3ea4e9da-91c8-4410-969d-49f53895860e nodeName:}" failed. No retries permitted until 2026-07-31 23:46:43.669821183 +0000 UTC m=+38.543464790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls") pod "image-registry-5cfbfcfdcd-tlc8x" (UID: "3ea4e9da-91c8-4410-969d-49f53895860e") : secret "image-registry-tls" not found
Jul 31 23:46:41.749258 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:41.749224    2584 generic.go:356] "Generic (PLEG): container finished" podID="054662f6-b4fc-47bf-afbc-dee3eec7a1a7" containerID="dfe209d9c028da0fede948e23668770c1c90acbb7ec9dd4ff0c70202eb331f0a" exitCode=0
Jul 31 23:46:41.749419 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:41.749296    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glccn" event={"ID":"054662f6-b4fc-47bf-afbc-dee3eec7a1a7","Type":"ContainerDied","Data":"dfe209d9c028da0fede948e23668770c1c90acbb7ec9dd4ff0c70202eb331f0a"}
Jul 31 23:46:41.770154 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:41.770083    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:41.770273 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:41.770257    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:41.770367 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:41.770314    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls podName:f6ce1e39-8710-41b1-8fe8-c6ba227ad006 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:43.770297149 +0000 UTC m=+38.643940752 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls") pod "dns-default-zmrgg" (UID: "f6ce1e39-8710-41b1-8fe8-c6ba227ad006") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:42.754539 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:42.754500    2584 generic.go:356] "Generic (PLEG): container finished" podID="054662f6-b4fc-47bf-afbc-dee3eec7a1a7" containerID="af93927fbe5bb158ae90d37106fa36eb0ace60506f20c4ad26e3d0c8d8aebbf7" exitCode=0
Jul 31 23:46:42.754539 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:42.754545    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glccn" event={"ID":"054662f6-b4fc-47bf-afbc-dee3eec7a1a7","Type":"ContainerDied","Data":"af93927fbe5bb158ae90d37106fa36eb0ace60506f20c4ad26e3d0c8d8aebbf7"}
Jul 31 23:46:43.683646 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:43.683613    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:43.683753 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:43.683737    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:43.683753 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:43.683749    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x: secret "image-registry-tls" not found
Jul 31 23:46:43.683869 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:43.683803    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls podName:3ea4e9da-91c8-4410-969d-49f53895860e nodeName:}" failed. No retries permitted until 2026-07-31 23:46:47.683788451 +0000 UTC m=+42.557432040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls") pod "image-registry-5cfbfcfdcd-tlc8x" (UID: "3ea4e9da-91c8-4410-969d-49f53895860e") : secret "image-registry-tls" not found
Jul 31 23:46:43.760442 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:43.760409    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glccn" event={"ID":"054662f6-b4fc-47bf-afbc-dee3eec7a1a7","Type":"ContainerStarted","Data":"b7c533991abe9bec4e9b5e9e74fa6e664f775e0ae88d61511c8db23fcd4d4f0c"}
Jul 31 23:46:43.761768 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:43.761742    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ghqfz" event={"ID":"af99387b-4f11-481c-a86d-aa666c795d94","Type":"ContainerStarted","Data":"c5d3dc6ac9957e4122b1e08822e97533d12fb613e316586341315f11b4a86885"}
Jul 31 23:46:43.783769 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:43.783640    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-glccn" podStartSLOduration=9.889842746 podStartE2EDuration="38.783627041s" totalImagesPullingTime="28.893784295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:43.781821805 +0000 UTC m=+38.655465415" watchObservedRunningTime="2026-07-31 23:46:43.783627041 +0000 UTC m=+38.657270651"
Jul 31 23:46:43.784379 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:43.784360    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:43.784518 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:43.784500    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:43.784582 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:43.784557    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls podName:f6ce1e39-8710-41b1-8fe8-c6ba227ad006 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:47.78454493 +0000 UTC m=+42.658188521 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls") pod "dns-default-zmrgg" (UID: "f6ce1e39-8710-41b1-8fe8-c6ba227ad006") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:43.798211 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:43.798161    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ghqfz" podStartSLOduration=2.6967203939999997 podStartE2EDuration="4.798147104s" totalImagesPullingTime="2.10142671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:43.797270485 +0000 UTC m=+38.670914095" watchObservedRunningTime="2026-07-31 23:46:43.798147104 +0000 UTC m=+38.671790713"
Jul 31 23:46:47.341641 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.341449    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-t6zcg"]
Jul 31 23:46:47.367888 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.367862    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-t6zcg"]
Jul 31 23:46:47.368049 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.368012    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-t6zcg"
Jul 31 23:46:47.370520 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.370501    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:47.370666 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.370543    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lzpxh\""
Jul 31 23:46:47.371583 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.371565    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:47.427442 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.427414    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97"]
Jul 31 23:46:47.448112 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.448087    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97"]
Jul 31 23:46:47.448253 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.448185    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97"
Jul 31 23:46:47.451558 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.451535    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:47.451890 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.451875    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 31 23:46:47.452436 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.452417    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:47.452672 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.452657    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-25khc\""
Jul 31 23:46:47.453105 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.453089    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 31 23:46:47.511346 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.511313    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9dxs\" (UniqueName: \"kubernetes.io/projected/c8df51d7-7157-4743-a324-f3ad06e16c81-kube-api-access-h9dxs\") pod \"volume-data-source-validator-5b8776669-t6zcg\" (UID: \"c8df51d7-7157-4743-a324-f3ad06e16c81\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-t6zcg"
Jul 31 23:46:47.534713 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.534684    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-b6g6c"]
Jul 31 23:46:47.556401 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.556369    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-rp75j"]
Jul 31 23:46:47.556563 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.556491    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:47.559293 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.559270    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:47.559418 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.559295    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:47.559418 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.559270    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 31 23:46:47.559586 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.559269    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4nts2\""
Jul 31 23:46:47.559705 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.559688    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 31 23:46:47.566503 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.566485    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 31 23:46:47.574465 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.574446    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-b6g6c"]
Jul 31 23:46:47.574554 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.574474    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-rp75j"]
Jul 31 23:46:47.574593 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.574554    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.577240 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.577199    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2vkwt\""
Jul 31 23:46:47.577361 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.577203    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 23:46:47.577457 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.577387    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 23:46:47.577546 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.577531    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 31 23:46:47.577630 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.577568    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 31 23:46:47.583622 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.583601    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 31 23:46:47.612016 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.611920    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9dxs\" (UniqueName: \"kubernetes.io/projected/c8df51d7-7157-4743-a324-f3ad06e16c81-kube-api-access-h9dxs\") pod \"volume-data-source-validator-5b8776669-t6zcg\" (UID: \"c8df51d7-7157-4743-a324-f3ad06e16c81\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-t6zcg"
Jul 31 23:46:47.612016 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.611995    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/51994692-c2a6-4b7e-a1c3-ca8255d407e8-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nmn97\" (UID: \"51994692-c2a6-4b7e-a1c3-ca8255d407e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97"
Jul 31 23:46:47.612175 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.612037    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51994692-c2a6-4b7e-a1c3-ca8255d407e8-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nmn97\" (UID: \"51994692-c2a6-4b7e-a1c3-ca8255d407e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97"
Jul 31 23:46:47.612175 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.612150    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ttkt\" (UniqueName: \"kubernetes.io/projected/51994692-c2a6-4b7e-a1c3-ca8255d407e8-kube-api-access-6ttkt\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nmn97\" (UID: \"51994692-c2a6-4b7e-a1c3-ca8255d407e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97"
Jul 31 23:46:47.631394 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.631370    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9dxs\" (UniqueName: \"kubernetes.io/projected/c8df51d7-7157-4743-a324-f3ad06e16c81-kube-api-access-h9dxs\") pod \"volume-data-source-validator-5b8776669-t6zcg\" (UID: \"c8df51d7-7157-4743-a324-f3ad06e16c81\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-t6zcg"
Jul 31 23:46:47.676892 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.676863    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-t6zcg"
Jul 31 23:46:47.713290 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.713267    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c31e0617-9006-4866-91ab-4a72295b5771-serving-cert\") pod \"console-operator-5ccd6b445f-b6g6c\" (UID: \"c31e0617-9006-4866-91ab-4a72295b5771\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:47.713404 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.713319    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/51994692-c2a6-4b7e-a1c3-ca8255d407e8-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nmn97\" (UID: \"51994692-c2a6-4b7e-a1c3-ca8255d407e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97"
Jul 31 23:46:47.713404 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.713362    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51994692-c2a6-4b7e-a1c3-ca8255d407e8-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nmn97\" (UID: \"51994692-c2a6-4b7e-a1c3-ca8255d407e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97"
Jul 31 23:46:47.713404 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.713391    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-serving-cert\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.713559 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.713422    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-service-ca-bundle\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.713559 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.713457    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c31e0617-9006-4866-91ab-4a72295b5771-config\") pod \"console-operator-5ccd6b445f-b6g6c\" (UID: \"c31e0617-9006-4866-91ab-4a72295b5771\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:47.713559 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.713491    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9z45\" (UniqueName: \"kubernetes.io/projected/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-kube-api-access-j9z45\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.713559 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.713522    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn88r\" (UniqueName: \"kubernetes.io/projected/c31e0617-9006-4866-91ab-4a72295b5771-kube-api-access-nn88r\") pod \"console-operator-5ccd6b445f-b6g6c\" (UID: \"c31e0617-9006-4866-91ab-4a72295b5771\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:47.713720 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.713584    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-snapshots\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.713720 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.713644    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:47.713720 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.713702    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-trusted-ca-bundle\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.713854 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.713730    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-tmp\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.713854 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:47.713741    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:47.713854 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:47.713758    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x: secret "image-registry-tls" not found
Jul 31 23:46:47.713854 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.713765    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c31e0617-9006-4866-91ab-4a72295b5771-trusted-ca\") pod \"console-operator-5ccd6b445f-b6g6c\" (UID: \"c31e0617-9006-4866-91ab-4a72295b5771\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:47.713854 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.713811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ttkt\" (UniqueName: \"kubernetes.io/projected/51994692-c2a6-4b7e-a1c3-ca8255d407e8-kube-api-access-6ttkt\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nmn97\" (UID: \"51994692-c2a6-4b7e-a1c3-ca8255d407e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97"
Jul 31 23:46:47.713854 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:47.713855    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls podName:3ea4e9da-91c8-4410-969d-49f53895860e nodeName:}" failed. No retries permitted until 2026-07-31 23:46:55.713835916 +0000 UTC m=+50.587479522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls") pod "image-registry-5cfbfcfdcd-tlc8x" (UID: "3ea4e9da-91c8-4410-969d-49f53895860e") : secret "image-registry-tls" not found
Jul 31 23:46:47.714141 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.713959    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51994692-c2a6-4b7e-a1c3-ca8255d407e8-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nmn97\" (UID: \"51994692-c2a6-4b7e-a1c3-ca8255d407e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97"
Jul 31 23:46:47.715636 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.715615    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/51994692-c2a6-4b7e-a1c3-ca8255d407e8-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nmn97\" (UID: \"51994692-c2a6-4b7e-a1c3-ca8255d407e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97"
Jul 31 23:46:47.721169 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.721150    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ttkt\" (UniqueName: \"kubernetes.io/projected/51994692-c2a6-4b7e-a1c3-ca8255d407e8-kube-api-access-6ttkt\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nmn97\" (UID: \"51994692-c2a6-4b7e-a1c3-ca8255d407e8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97"
Jul 31 23:46:47.756787 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.756761    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97"
Jul 31 23:46:47.815393 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.815354    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-serving-cert\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.815563 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.815412    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-service-ca-bundle\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.815563 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.815447    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c31e0617-9006-4866-91ab-4a72295b5771-config\") pod \"console-operator-5ccd6b445f-b6g6c\" (UID: \"c31e0617-9006-4866-91ab-4a72295b5771\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:47.815563 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.815480    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9z45\" (UniqueName: \"kubernetes.io/projected/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-kube-api-access-j9z45\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.815563 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.815517    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn88r\" (UniqueName: \"kubernetes.io/projected/c31e0617-9006-4866-91ab-4a72295b5771-kube-api-access-nn88r\") pod \"console-operator-5ccd6b445f-b6g6c\" (UID: \"c31e0617-9006-4866-91ab-4a72295b5771\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:47.815563 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.815550    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-snapshots\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.815826 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.815581    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:47.815826 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.815638    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-trusted-ca-bundle\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.815826 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.815663    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-tmp\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.815826 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.815698    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c31e0617-9006-4866-91ab-4a72295b5771-trusted-ca\") pod \"console-operator-5ccd6b445f-b6g6c\" (UID: \"c31e0617-9006-4866-91ab-4a72295b5771\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:47.815826 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.815728    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c31e0617-9006-4866-91ab-4a72295b5771-serving-cert\") pod \"console-operator-5ccd6b445f-b6g6c\" (UID: \"c31e0617-9006-4866-91ab-4a72295b5771\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:47.816780 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.816231    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-service-ca-bundle\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.816780 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.816344    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-t6zcg"]
Jul 31 23:46:47.816780 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.816475    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-tmp\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.816780 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.816687    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c31e0617-9006-4866-91ab-4a72295b5771-config\") pod \"console-operator-5ccd6b445f-b6g6c\" (UID: \"c31e0617-9006-4866-91ab-4a72295b5771\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:47.816780 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:47.816784    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:47.817120 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:47.816856    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls podName:f6ce1e39-8710-41b1-8fe8-c6ba227ad006 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:55.816836355 +0000 UTC m=+50.690479945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls") pod "dns-default-zmrgg" (UID: "f6ce1e39-8710-41b1-8fe8-c6ba227ad006") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:47.817120 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.816996    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-snapshots\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.817120 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.817028    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-trusted-ca-bundle\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.817120 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.817028    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c31e0617-9006-4866-91ab-4a72295b5771-trusted-ca\") pod \"console-operator-5ccd6b445f-b6g6c\" (UID: \"c31e0617-9006-4866-91ab-4a72295b5771\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:47.818755 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.818724    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c31e0617-9006-4866-91ab-4a72295b5771-serving-cert\") pod \"console-operator-5ccd6b445f-b6g6c\" (UID: \"c31e0617-9006-4866-91ab-4a72295b5771\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:47.819090 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.819069    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-serving-cert\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.819847 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:47.819826    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8df51d7_7157_4743_a324_f3ad06e16c81.slice/crio-cfe69f90951a1b8ad929de26e11a33d55ac1d3987713b82b7adbba95b261f390 WatchSource:0}: Error finding container cfe69f90951a1b8ad929de26e11a33d55ac1d3987713b82b7adbba95b261f390: Status 404 returned error can't find the container with id cfe69f90951a1b8ad929de26e11a33d55ac1d3987713b82b7adbba95b261f390
Jul 31 23:46:47.823853 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.823827    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn88r\" (UniqueName: \"kubernetes.io/projected/c31e0617-9006-4866-91ab-4a72295b5771-kube-api-access-nn88r\") pod \"console-operator-5ccd6b445f-b6g6c\" (UID: \"c31e0617-9006-4866-91ab-4a72295b5771\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:47.823938 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.823853    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9z45\" (UniqueName: \"kubernetes.io/projected/45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b-kube-api-access-j9z45\") pod \"insights-operator-77696685b-rp75j\" (UID: \"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b\") " pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.867455 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.867380    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:47.883717 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.883688    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-rp75j"
Jul 31 23:46:47.896404 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:47.896377    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97"]
Jul 31 23:46:47.900257 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:47.900235    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51994692_c2a6_4b7e_a1c3_ca8255d407e8.slice/crio-a2199af649c0616775ac0bcba3a0cac750224d929d370e5074d3d71cd09f27af WatchSource:0}: Error finding container a2199af649c0616775ac0bcba3a0cac750224d929d370e5074d3d71cd09f27af: Status 404 returned error can't find the container with id a2199af649c0616775ac0bcba3a0cac750224d929d370e5074d3d71cd09f27af
Jul 31 23:46:48.026812 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.026778    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-b6g6c"]
Jul 31 23:46:48.036117 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:48.036091    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31e0617_9006_4866_91ab_4a72295b5771.slice/crio-68e43d688876e8d5b339078ca226b085a4fd516c78e30fb15bf5cc660da71454 WatchSource:0}: Error finding container 68e43d688876e8d5b339078ca226b085a4fd516c78e30fb15bf5cc660da71454: Status 404 returned error can't find the container with id 68e43d688876e8d5b339078ca226b085a4fd516c78e30fb15bf5cc660da71454
Jul 31 23:46:48.059451 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.059430    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-rp75j"]
Jul 31 23:46:48.061476 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:46:48.061455    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45edc69d_fbf6_46bf_8c4b_9aecc7c2f43b.slice/crio-e911816f7d6bf4e9ddf97ab5c7cc9ea6c2d7d96755d647f4a66b2dffef7c4483 WatchSource:0}: Error finding container e911816f7d6bf4e9ddf97ab5c7cc9ea6c2d7d96755d647f4a66b2dffef7c4483: Status 404 returned error can't find the container with id e911816f7d6bf4e9ddf97ab5c7cc9ea6c2d7d96755d647f4a66b2dffef7c4483
Jul 31 23:46:48.199786 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.199708    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"]
Jul 31 23:46:48.219284 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.219260    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"]
Jul 31 23:46:48.219407 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.219348    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"
Jul 31 23:46:48.222072 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.222053    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 31 23:46:48.222500 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.222485    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 31 23:46:48.222605 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.222589    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6mfmh\""
Jul 31 23:46:48.320525 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.320491    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-wpznf\" (UID: \"65c4d1ec-7577-4e31-bc3e-cfed4f74a062\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"
Jul 31 23:46:48.320672 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.320538    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-wpznf\" (UID: \"65c4d1ec-7577-4e31-bc3e-cfed4f74a062\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"
Jul 31 23:46:48.421284 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.421245    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-wpznf\" (UID: \"65c4d1ec-7577-4e31-bc3e-cfed4f74a062\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"
Jul 31 23:46:48.421678 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.421361    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-wpznf\" (UID: \"65c4d1ec-7577-4e31-bc3e-cfed4f74a062\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"
Jul 31 23:46:48.421678 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:48.421482    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:48.421678 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:48.421536    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert podName:65c4d1ec-7577-4e31-bc3e-cfed4f74a062 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:48.921522583 +0000 UTC m=+43.795166172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-wpznf" (UID: "65c4d1ec-7577-4e31-bc3e-cfed4f74a062") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:48.421845 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.421829    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-wpznf\" (UID: \"65c4d1ec-7577-4e31-bc3e-cfed4f74a062\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"
Jul 31 23:46:48.781512 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.781413    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c" event={"ID":"c31e0617-9006-4866-91ab-4a72295b5771","Type":"ContainerStarted","Data":"68e43d688876e8d5b339078ca226b085a4fd516c78e30fb15bf5cc660da71454"}
Jul 31 23:46:48.783191 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.783119    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97" event={"ID":"51994692-c2a6-4b7e-a1c3-ca8255d407e8","Type":"ContainerStarted","Data":"a2199af649c0616775ac0bcba3a0cac750224d929d370e5074d3d71cd09f27af"}
Jul 31 23:46:48.784895 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.784868    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-t6zcg" event={"ID":"c8df51d7-7157-4743-a324-f3ad06e16c81","Type":"ContainerStarted","Data":"cfe69f90951a1b8ad929de26e11a33d55ac1d3987713b82b7adbba95b261f390"}
Jul 31 23:46:48.787008 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.786983    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-rp75j" event={"ID":"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b","Type":"ContainerStarted","Data":"e911816f7d6bf4e9ddf97ab5c7cc9ea6c2d7d96755d647f4a66b2dffef7c4483"}
Jul 31 23:46:48.926749 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:48.926709    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-wpznf\" (UID: \"65c4d1ec-7577-4e31-bc3e-cfed4f74a062\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"
Jul 31 23:46:48.926987 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:48.926940    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:48.927080 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:48.927040    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert podName:65c4d1ec-7577-4e31-bc3e-cfed4f74a062 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:49.927016441 +0000 UTC m=+44.800660049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-wpznf" (UID: "65c4d1ec-7577-4e31-bc3e-cfed4f74a062") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:49.791343 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:49.791246    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-t6zcg" event={"ID":"c8df51d7-7157-4743-a324-f3ad06e16c81","Type":"ContainerStarted","Data":"65378f0f63caaf0be193a1ee1527b279ac7758d162a4bd6764fd1d2cb756f61a"}
Jul 31 23:46:49.807485 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:49.807434    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-t6zcg" podStartSLOduration=1.3258929130000001 podStartE2EDuration="2.80741751s" totalImagesPullingTime="1.481524597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:49.806522345 +0000 UTC m=+44.680165955" watchObservedRunningTime="2026-07-31 23:46:49.80741751 +0000 UTC m=+44.681061119"
Jul 31 23:46:49.935856 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:49.935815    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-wpznf\" (UID: \"65c4d1ec-7577-4e31-bc3e-cfed4f74a062\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"
Jul 31 23:46:49.936044 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:49.935988    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:49.936087 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:49.936062    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert podName:65c4d1ec-7577-4e31-bc3e-cfed4f74a062 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:51.936044592 +0000 UTC m=+46.809688180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-wpznf" (UID: "65c4d1ec-7577-4e31-bc3e-cfed4f74a062") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:51.797389 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:51.797356    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97" event={"ID":"51994692-c2a6-4b7e-a1c3-ca8255d407e8","Type":"ContainerStarted","Data":"e6a7ba6826d4da93ef9c843d7401795bff31b1252df926e7d3b2fe08036e5600"}
Jul 31 23:46:51.952333 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:51.952290    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-wpznf\" (UID: \"65c4d1ec-7577-4e31-bc3e-cfed4f74a062\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"
Jul 31 23:46:51.952496 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:51.952463    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:51.952572 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:51.952548    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert podName:65c4d1ec-7577-4e31-bc3e-cfed4f74a062 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:55.952526168 +0000 UTC m=+50.826169760 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-wpznf" (UID: "65c4d1ec-7577-4e31-bc3e-cfed4f74a062") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:52.801491 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:52.801452    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-rp75j" event={"ID":"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b","Type":"ContainerStarted","Data":"cc92992690bbc15d380af5d942cdf4441bfa7914e75ccb36f560221112cc0a7b"}
Jul 31 23:46:52.803118 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:52.803098    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/0.log"
Jul 31 23:46:52.803248 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:52.803134    2584 generic.go:356] "Generic (PLEG): container finished" podID="c31e0617-9006-4866-91ab-4a72295b5771" containerID="dcbf65dce342957ae8be95193a3bd642100eadc68258f3c07033d4bf13015891" exitCode=255
Jul 31 23:46:52.803248 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:52.803203    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c" event={"ID":"c31e0617-9006-4866-91ab-4a72295b5771","Type":"ContainerDied","Data":"dcbf65dce342957ae8be95193a3bd642100eadc68258f3c07033d4bf13015891"}
Jul 31 23:46:52.803387 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:52.803373    2584 scope.go:117] "RemoveContainer" containerID="dcbf65dce342957ae8be95193a3bd642100eadc68258f3c07033d4bf13015891"
Jul 31 23:46:52.826070 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:52.826013    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-rp75j" podStartSLOduration=2.183752752 podStartE2EDuration="5.825990289s" totalImagesPullingTime="3.642237537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:52.824107548 +0000 UTC m=+47.697751159" watchObservedRunningTime="2026-07-31 23:46:52.825990289 +0000 UTC m=+47.699633900"
Jul 31 23:46:52.854610 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:52.854559    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97" podStartSLOduration=2.055897953 podStartE2EDuration="5.854538955s" totalImagesPullingTime="3.798641002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:52.854132079 +0000 UTC m=+47.727775690" watchObservedRunningTime="2026-07-31 23:46:52.854538955 +0000 UTC m=+47.728182566"
Jul 31 23:46:53.307322 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.307287    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x799c"]
Jul 31 23:46:53.334894 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.334853    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x799c"]
Jul 31 23:46:53.335087 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.335005    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:53.338587 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.338564    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 23:46:53.338734 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.338564    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 23:46:53.338734 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.338629    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4rmhk\""
Jul 31 23:46:53.465506 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.465462    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-crio-socket\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:53.465506 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.465503    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-data-volume\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:53.465764 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.465548    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv89c\" (UniqueName: \"kubernetes.io/projected/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-kube-api-access-xv89c\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:53.465764 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.465613    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:53.465764 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.465670    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:53.567100 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.567001    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv89c\" (UniqueName: \"kubernetes.io/projected/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-kube-api-access-xv89c\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:53.567260 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.567165    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:53.567260 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.567239    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:53.567396 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.567303    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-crio-socket\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:53.567396 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:53.567310    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 23:46:53.567396 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.567327    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-data-volume\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:53.567396 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:53.567388    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-kube-rbac-proxy-cm podName:59ff7f6d-a786-4afb-9fd6-90b59a73ffd9 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:54.067367753 +0000 UTC m=+48.941011354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x799c" (UID: "59ff7f6d-a786-4afb-9fd6-90b59a73ffd9") : configmap "kube-rbac-proxy" not found
Jul 31 23:46:53.567604 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:53.567398    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:53.567604 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:53.567441    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls podName:59ff7f6d-a786-4afb-9fd6-90b59a73ffd9 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:54.06742962 +0000 UTC m=+48.941073212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x799c" (UID: "59ff7f6d-a786-4afb-9fd6-90b59a73ffd9") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:53.567604 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.567529    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-crio-socket\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:53.567756 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.567689    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-data-volume\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:53.576647 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.576621    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv89c\" (UniqueName: \"kubernetes.io/projected/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-kube-api-access-xv89c\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:53.808114 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.808083    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Jul 31 23:46:53.808524 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.808488    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/0.log"
Jul 31 23:46:53.808584 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.808532    2584 generic.go:356] "Generic (PLEG): container finished" podID="c31e0617-9006-4866-91ab-4a72295b5771" containerID="b1620ac4fb089e0cc495741919f4c1641508adc3e6398b7da224f3f6dd0066a3" exitCode=255
Jul 31 23:46:53.808643 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.808621    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c" event={"ID":"c31e0617-9006-4866-91ab-4a72295b5771","Type":"ContainerDied","Data":"b1620ac4fb089e0cc495741919f4c1641508adc3e6398b7da224f3f6dd0066a3"}
Jul 31 23:46:53.808690 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.808677    2584 scope.go:117] "RemoveContainer" containerID="dcbf65dce342957ae8be95193a3bd642100eadc68258f3c07033d4bf13015891"
Jul 31 23:46:53.808868 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:53.808852    2584 scope.go:117] "RemoveContainer" containerID="b1620ac4fb089e0cc495741919f4c1641508adc3e6398b7da224f3f6dd0066a3"
Jul 31 23:46:53.809077 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:53.809049    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-b6g6c_openshift-console-operator(c31e0617-9006-4866-91ab-4a72295b5771)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c" podUID="c31e0617-9006-4866-91ab-4a72295b5771"
Jul 31 23:46:54.072470 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:54.072429    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:54.072642 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:54.072497    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:54.072642 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:54.072562    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 23:46:54.072642 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:54.072594    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:54.072642 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:54.072638    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-kube-rbac-proxy-cm podName:59ff7f6d-a786-4afb-9fd6-90b59a73ffd9 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:55.072619511 +0000 UTC m=+49.946263120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x799c" (UID: "59ff7f6d-a786-4afb-9fd6-90b59a73ffd9") : configmap "kube-rbac-proxy" not found
Jul 31 23:46:54.072801 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:54.072655    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls podName:59ff7f6d-a786-4afb-9fd6-90b59a73ffd9 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:55.072649364 +0000 UTC m=+49.946292952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x799c" (UID: "59ff7f6d-a786-4afb-9fd6-90b59a73ffd9") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:54.812498 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:54.812470    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Jul 31 23:46:54.812862 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:54.812787    2584 scope.go:117] "RemoveContainer" containerID="b1620ac4fb089e0cc495741919f4c1641508adc3e6398b7da224f3f6dd0066a3"
Jul 31 23:46:54.812942 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:54.812925    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-b6g6c_openshift-console-operator(c31e0617-9006-4866-91ab-4a72295b5771)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c" podUID="c31e0617-9006-4866-91ab-4a72295b5771"
Jul 31 23:46:55.081188 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:55.081097    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:55.081328 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:55.081263    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:55.081328 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:55.081317    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 23:46:55.081399 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:55.081328    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls podName:59ff7f6d-a786-4afb-9fd6-90b59a73ffd9 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:57.081312797 +0000 UTC m=+51.954956390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x799c" (UID: "59ff7f6d-a786-4afb-9fd6-90b59a73ffd9") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:55.081399 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:55.081274    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:55.081399 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:55.081359    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-kube-rbac-proxy-cm podName:59ff7f6d-a786-4afb-9fd6-90b59a73ffd9 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:57.081348183 +0000 UTC m=+51.954991771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x799c" (UID: "59ff7f6d-a786-4afb-9fd6-90b59a73ffd9") : configmap "kube-rbac-proxy" not found
Jul 31 23:46:55.786713 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:55.786675    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:46:55.786903 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:55.786831    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:55.786903 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:55.786851    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x: secret "image-registry-tls" not found
Jul 31 23:46:55.787018 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:55.786908    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls podName:3ea4e9da-91c8-4410-969d-49f53895860e nodeName:}" failed. No retries permitted until 2026-07-31 23:47:11.786891619 +0000 UTC m=+66.660535212 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls") pod "image-registry-5cfbfcfdcd-tlc8x" (UID: "3ea4e9da-91c8-4410-969d-49f53895860e") : secret "image-registry-tls" not found
Jul 31 23:46:55.823122 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:55.823096    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hsjmv_1df3a7f4-80f0-4a56-a857-d59531130023/dns-node-resolver/0.log"
Jul 31 23:46:55.887723 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:55.887688    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:46:55.887888 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:55.887844    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:55.887926 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:55.887906    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls podName:f6ce1e39-8710-41b1-8fe8-c6ba227ad006 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:11.887888819 +0000 UTC m=+66.761532411 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls") pod "dns-default-zmrgg" (UID: "f6ce1e39-8710-41b1-8fe8-c6ba227ad006") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:55.988354 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:55.988315    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-wpznf\" (UID: \"65c4d1ec-7577-4e31-bc3e-cfed4f74a062\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"
Jul 31 23:46:55.988556 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:55.988501    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:55.988619 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:55.988583    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert podName:65c4d1ec-7577-4e31-bc3e-cfed4f74a062 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:03.988561534 +0000 UTC m=+58.862205127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-wpznf" (UID: "65c4d1ec-7577-4e31-bc3e-cfed4f74a062") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:57.096388 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:57.096352    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:57.096774 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:57.096415    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:57.096774 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:57.096523    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:57.096774 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:57.096580    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls podName:59ff7f6d-a786-4afb-9fd6-90b59a73ffd9 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:01.096564621 +0000 UTC m=+55.970208215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x799c" (UID: "59ff7f6d-a786-4afb-9fd6-90b59a73ffd9") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:57.096894 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:57.096867    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:46:57.227920 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:57.227893    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s9w8v_ce48bcb1-ccef-4183-af51-8a88c043cb64/node-ca/0.log"
Jul 31 23:46:57.821199 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:57.821171    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ghqfz_af99387b-4f11-481c-a86d-aa666c795d94/serve-healthcheck-canary/0.log"
Jul 31 23:46:57.868492 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:57.868445    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:57.868492 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:57.868499    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:46:57.868841 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:57.868829    2584 scope.go:117] "RemoveContainer" containerID="b1620ac4fb089e0cc495741919f4c1641508adc3e6398b7da224f3f6dd0066a3"
Jul 31 23:46:57.869003 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:46:57.868984    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-b6g6c_openshift-console-operator(c31e0617-9006-4866-91ab-4a72295b5771)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c" podUID="c31e0617-9006-4866-91ab-4a72295b5771"
Jul 31 23:46:58.628877 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:46:58.628841    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-nmn97_51994692-c2a6-4b7e-a1c3-ca8255d407e8/kube-storage-version-migrator-operator/0.log"
Jul 31 23:47:01.128281 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:01.128240    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:47:01.128770 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:47:01.128400    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:47:01.128770 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:47:01.128475    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls podName:59ff7f6d-a786-4afb-9fd6-90b59a73ffd9 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:09.128460146 +0000 UTC m=+64.002103734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x799c" (UID: "59ff7f6d-a786-4afb-9fd6-90b59a73ffd9") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:47:04.050946 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:04.050906    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-wpznf\" (UID: \"65c4d1ec-7577-4e31-bc3e-cfed4f74a062\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"
Jul 31 23:47:04.051339 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:47:04.051050    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:47:04.051339 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:47:04.051110    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert podName:65c4d1ec-7577-4e31-bc3e-cfed4f74a062 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:20.051095548 +0000 UTC m=+74.924739135 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-wpznf" (UID: "65c4d1ec-7577-4e31-bc3e-cfed4f74a062") : secret "networking-console-plugin-cert" not found
Jul 31 23:47:05.743351 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:05.743317    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khzpq"
Jul 31 23:47:09.193445 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:09.193392    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:47:09.195713 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:09.195682    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59ff7f6d-a786-4afb-9fd6-90b59a73ffd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x799c\" (UID: \"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9\") " pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:47:09.246636 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:09.246613    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4rmhk\""
Jul 31 23:47:09.254725 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:09.254707    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x799c"
Jul 31 23:47:09.373712 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:09.373684    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x799c"]
Jul 31 23:47:09.377588 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:47:09.377563    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ff7f6d_a786_4afb_9fd6_90b59a73ffd9.slice/crio-6634077f95e2d7e3a0d29ff73f08bfc21b08fc265edc084685cc926169e78db8 WatchSource:0}: Error finding container 6634077f95e2d7e3a0d29ff73f08bfc21b08fc265edc084685cc926169e78db8: Status 404 returned error can't find the container with id 6634077f95e2d7e3a0d29ff73f08bfc21b08fc265edc084685cc926169e78db8
Jul 31 23:47:09.566316 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:09.566294    2584 scope.go:117] "RemoveContainer" containerID="b1620ac4fb089e0cc495741919f4c1641508adc3e6398b7da224f3f6dd0066a3"
Jul 31 23:47:09.853645 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:09.853558    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x799c" event={"ID":"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9","Type":"ContainerStarted","Data":"1080801a339a8d253d02fc14c294d909f85144fea91a9200a52f02b2ba74dee6"}
Jul 31 23:47:09.853645 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:09.853628    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x799c" event={"ID":"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9","Type":"ContainerStarted","Data":"6634077f95e2d7e3a0d29ff73f08bfc21b08fc265edc084685cc926169e78db8"}
Jul 31 23:47:09.855605 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:09.855584    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Jul 31 23:47:09.855742 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:09.855642    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c" event={"ID":"c31e0617-9006-4866-91ab-4a72295b5771","Type":"ContainerStarted","Data":"7488511d646efc0c4dbb8b6536d8d4a7b14c27e4b52729cf31d2b36dbe1cfa28"}
Jul 31 23:47:09.856016 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:09.855951    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:47:09.877612 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:09.877567    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c" podStartSLOduration=19.214849144 podStartE2EDuration="22.8775481s" totalImagesPullingTime="3.662698956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:09.876714683 +0000 UTC m=+64.750358295" watchObservedRunningTime="2026-07-31 23:47:09.8775481 +0000 UTC m=+64.751191688"
Jul 31 23:47:10.479027 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:10.479000    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-b6g6c"
Jul 31 23:47:10.861099 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:10.861023    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x799c" event={"ID":"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9","Type":"ContainerStarted","Data":"d5bc1aae697e64ff57c53985842d3fad8ec00a55cd75d08558dbdd99d0027cf9"}
Jul 31 23:47:11.314129 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.314094    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs\") pod \"network-metrics-daemon-hsnnq\" (UID: \"9d3952c7-cede-4f37-aba3-9b96b84eaeb6\") " pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:47:11.316734 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.316711    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 23:47:11.327207 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.327186    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3952c7-cede-4f37-aba3-9b96b84eaeb6-metrics-certs\") pod \"network-metrics-daemon-hsnnq\" (UID: \"9d3952c7-cede-4f37-aba3-9b96b84eaeb6\") " pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:47:11.415287 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.415261    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:47:11.418217 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.418198    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 23:47:11.428317 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.428292    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b717d75a-de98-4e7b-b86e-1af68be8f6a7-original-pull-secret\") pod \"global-pull-secret-syncer-sjj4w\" (UID: \"b717d75a-de98-4e7b-b86e-1af68be8f6a7\") " pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:47:11.516155 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.516118    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2f6w\" (UniqueName: \"kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w\") pod \"network-check-target-qpmwv\" (UID: \"6ca51531-7776-413a-98b3-aebadd923730\") " pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:47:11.518508 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.518491    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 23:47:11.528677 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.528660    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 23:47:11.538760 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.538732    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2f6w\" (UniqueName: \"kubernetes.io/projected/6ca51531-7776-413a-98b3-aebadd923730-kube-api-access-h2f6w\") pod \"network-check-target-qpmwv\" (UID: \"6ca51531-7776-413a-98b3-aebadd923730\") " pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:47:11.578068 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.578026    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjj4w"
Jul 31 23:47:11.586664 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.586648    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n5wdr\""
Jul 31 23:47:11.591780 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.591762    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zhtdr\""
Jul 31 23:47:11.594705 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.594685    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:47:11.600328 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.600227    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsnnq"
Jul 31 23:47:11.722148 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.722110    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sjj4w"]
Jul 31 23:47:11.725130 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:47:11.725102    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb717d75a_de98_4e7b_b86e_1af68be8f6a7.slice/crio-dcf065fdb34a2cabd56367592f0d109bebc20a407ccfc2afa1f75d2bb57b961c WatchSource:0}: Error finding container dcf065fdb34a2cabd56367592f0d109bebc20a407ccfc2afa1f75d2bb57b961c: Status 404 returned error can't find the container with id dcf065fdb34a2cabd56367592f0d109bebc20a407ccfc2afa1f75d2bb57b961c
Jul 31 23:47:11.819142 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.819117    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:47:11.821282 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.821263    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls\") pod \"image-registry-5cfbfcfdcd-tlc8x\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") " pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:47:11.865400 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.865336    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x799c" event={"ID":"59ff7f6d-a786-4afb-9fd6-90b59a73ffd9","Type":"ContainerStarted","Data":"dca707737fc59deb59aa8782be6a265df2824f8333491b9a1eaa4baef609cc41"}
Jul 31 23:47:11.866480 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.866460    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sjj4w" event={"ID":"b717d75a-de98-4e7b-b86e-1af68be8f6a7","Type":"ContainerStarted","Data":"dcf065fdb34a2cabd56367592f0d109bebc20a407ccfc2afa1f75d2bb57b961c"}
Jul 31 23:47:11.888435 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.888391    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x799c" podStartSLOduration=16.99684728 podStartE2EDuration="18.888377874s" totalImagesPullingTime="1.891530594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:11.888280362 +0000 UTC m=+66.761923996" watchObservedRunningTime="2026-07-31 23:47:11.888377874 +0000 UTC m=+66.762021484"
Jul 31 23:47:11.919695 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.919665    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:47:11.922032 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.922006    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ce1e39-8710-41b1-8fe8-c6ba227ad006-metrics-tls\") pod \"dns-default-zmrgg\" (UID: \"f6ce1e39-8710-41b1-8fe8-c6ba227ad006\") " pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:47:11.947535 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.947466    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hsnnq"]
Jul 31 23:47:11.950135 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:47:11.950114    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3952c7_cede_4f37_aba3_9b96b84eaeb6.slice/crio-900eb28e03e9f7aa700c51d2724e29acde09ca19c418315d594ef67a9f27f0fd WatchSource:0}: Error finding container 900eb28e03e9f7aa700c51d2724e29acde09ca19c418315d594ef67a9f27f0fd: Status 404 returned error can't find the container with id 900eb28e03e9f7aa700c51d2724e29acde09ca19c418315d594ef67a9f27f0fd
Jul 31 23:47:11.950304 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:11.950285    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qpmwv"]
Jul 31 23:47:11.953005 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:47:11.952982    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca51531_7776_413a_98b3_aebadd923730.slice/crio-2563d2df3862990a6912a0aa405497dcb9e46df29dc78f6690fa200900ec2eb3 WatchSource:0}: Error finding container 2563d2df3862990a6912a0aa405497dcb9e46df29dc78f6690fa200900ec2eb3: Status 404 returned error can't find the container with id 2563d2df3862990a6912a0aa405497dcb9e46df29dc78f6690fa200900ec2eb3
Jul 31 23:47:12.090125 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:12.090094    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mbvf9\""
Jul 31 23:47:12.098452 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:12.098434    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:47:12.110129 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:12.110106    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r2mdc\""
Jul 31 23:47:12.118205 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:12.118157    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:47:12.272130 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:12.272108    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"]
Jul 31 23:47:12.275549 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:47:12.275516    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea4e9da_91c8_4410_969d_49f53895860e.slice/crio-edee64e17ab6fbab28116db3735e274a209c31ca6bdb87044af8ccab6a5cb088 WatchSource:0}: Error finding container edee64e17ab6fbab28116db3735e274a209c31ca6bdb87044af8ccab6a5cb088: Status 404 returned error can't find the container with id edee64e17ab6fbab28116db3735e274a209c31ca6bdb87044af8ccab6a5cb088
Jul 31 23:47:12.304042 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:12.304016    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zmrgg"]
Jul 31 23:47:12.307068 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:47:12.307020    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ce1e39_8710_41b1_8fe8_c6ba227ad006.slice/crio-5ff64ab0a5b073c85891294b37e493198fbc143d7a9b28868ff839a9ee89820c WatchSource:0}: Error finding container 5ff64ab0a5b073c85891294b37e493198fbc143d7a9b28868ff839a9ee89820c: Status 404 returned error can't find the container with id 5ff64ab0a5b073c85891294b37e493198fbc143d7a9b28868ff839a9ee89820c
Jul 31 23:47:12.875480 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:12.874737    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x" event={"ID":"3ea4e9da-91c8-4410-969d-49f53895860e","Type":"ContainerStarted","Data":"50f38e164e144fceaba3828cddf7f9d89b25f865b732299f5c633fdb876e8287"}
Jul 31 23:47:12.875480 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:12.874797    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x" event={"ID":"3ea4e9da-91c8-4410-969d-49f53895860e","Type":"ContainerStarted","Data":"edee64e17ab6fbab28116db3735e274a209c31ca6bdb87044af8ccab6a5cb088"}
Jul 31 23:47:12.876024 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:12.875573    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:47:12.878120 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:12.878034    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hsnnq" event={"ID":"9d3952c7-cede-4f37-aba3-9b96b84eaeb6","Type":"ContainerStarted","Data":"900eb28e03e9f7aa700c51d2724e29acde09ca19c418315d594ef67a9f27f0fd"}
Jul 31 23:47:12.881426 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:12.881399    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmrgg" event={"ID":"f6ce1e39-8710-41b1-8fe8-c6ba227ad006","Type":"ContainerStarted","Data":"5ff64ab0a5b073c85891294b37e493198fbc143d7a9b28868ff839a9ee89820c"}
Jul 31 23:47:12.884453 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:12.884416    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qpmwv" event={"ID":"6ca51531-7776-413a-98b3-aebadd923730","Type":"ContainerStarted","Data":"2563d2df3862990a6912a0aa405497dcb9e46df29dc78f6690fa200900ec2eb3"}
Jul 31 23:47:12.897462 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:12.897420    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x" podStartSLOduration=67.897407165 podStartE2EDuration="1m7.897407165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:12.897010236 +0000 UTC m=+67.770653843" watchObservedRunningTime="2026-07-31 23:47:12.897407165 +0000 UTC m=+67.771050775"
Jul 31 23:47:13.891561 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:13.891519    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hsnnq" event={"ID":"9d3952c7-cede-4f37-aba3-9b96b84eaeb6","Type":"ContainerStarted","Data":"ef5adcf16e775c4e4ef291cffc29b7b73a4ea8f71e25e32ee754a323f2f746b8"}
Jul 31 23:47:15.863537 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.863505    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"]
Jul 31 23:47:15.866501 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.866476    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:15.869093 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.869073    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 31 23:47:15.869229 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.869205    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 31 23:47:15.869229 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.869216    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 31 23:47:15.870246 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.870223    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 31 23:47:15.870376 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.870267    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 31 23:47:15.870376 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.870267    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 31 23:47:15.870376 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.870359    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 31 23:47:15.883134 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.883105    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"]
Jul 31 23:47:15.956320 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.956280    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-hub\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:15.956490 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.956342    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:15.956490 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.956390    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjqfg\" (UniqueName: \"kubernetes.io/projected/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-kube-api-access-vjqfg\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:15.956490 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.956419    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-ca\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:15.956490 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.956443    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-additional-service-ca\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:15.956490 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:15.956474    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:16.057919 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.057889    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-hub\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:16.058085 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.057948    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:16.058085 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.058011    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjqfg\" (UniqueName: \"kubernetes.io/projected/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-kube-api-access-vjqfg\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:16.058085 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.058041    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-ca\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:16.058085 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.058066    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-additional-service-ca\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:16.058292 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.058189    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:16.058916 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.058889    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-additional-service-ca\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:16.061208 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.061065    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-ca\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:16.061208 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.061129    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:16.061208 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.061140    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-hub\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:16.061208 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.061149    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:16.069481 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.069455    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjqfg\" (UniqueName: \"kubernetes.io/projected/19dfca1c-cd68-4742-a4cc-ee7ce492ff30-kube-api-access-vjqfg\") pod \"cluster-proxy-proxy-agent-664b6bf69b-dkf5w\" (UID: \"19dfca1c-cd68-4742-a4cc-ee7ce492ff30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:16.184903 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.184820    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"
Jul 31 23:47:16.903630 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.903598    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qpmwv" event={"ID":"6ca51531-7776-413a-98b3-aebadd923730","Type":"ContainerStarted","Data":"8e317c9c6a413401fadd99b03252c04fa4489b04a15cf50b6ce3cbe7d08de50f"}
Jul 31 23:47:16.904021 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.903698    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:47:16.905590 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.905561    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hsnnq" event={"ID":"9d3952c7-cede-4f37-aba3-9b96b84eaeb6","Type":"ContainerStarted","Data":"13debf6833aeba3da502a25f0421ee01e1d9a40b6cc32ca80575981336006879"}
Jul 31 23:47:16.922846 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.922714    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qpmwv" podStartSLOduration=66.122068384 podStartE2EDuration="1m10.922694098s" totalImagesPullingTime="4.800625714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:16.921298375 +0000 UTC m=+71.794941986" watchObservedRunningTime="2026-07-31 23:47:16.922694098 +0000 UTC m=+71.796337709"
Jul 31 23:47:16.932180 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.931806    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w"]
Jul 31 23:47:16.943551 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:16.943179    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hsnnq" podStartSLOduration=70.357790379 podStartE2EDuration="1m11.943158872s" totalImagesPullingTime="1.585368493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:16.941875557 +0000 UTC m=+71.815519169" watchObservedRunningTime="2026-07-31 23:47:16.943158872 +0000 UTC m=+71.816802482"
Jul 31 23:47:17.911674 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:17.911636    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmrgg" event={"ID":"f6ce1e39-8710-41b1-8fe8-c6ba227ad006","Type":"ContainerStarted","Data":"07e529d4b5613f7bb1b1a7076499e5e87be4a1713c2162acc5bae1f6037e08f6"}
Jul 31 23:47:17.912165 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:17.911841    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:47:17.912165 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:17.911889    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmrgg" event={"ID":"f6ce1e39-8710-41b1-8fe8-c6ba227ad006","Type":"ContainerStarted","Data":"3ab8546a5a9a2ba190900b1d97b9dc51bc03c9685717baea3b631a149dcef51b"}
Jul 31 23:47:17.913077 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:17.913039    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w" event={"ID":"19dfca1c-cd68-4742-a4cc-ee7ce492ff30","Type":"ContainerStarted","Data":"679571941c372927712b73607b2a1f7030fe2fd171258e42f25c72d6ca2f29f2"}
Jul 31 23:47:17.914582 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:17.914551    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sjj4w" event={"ID":"b717d75a-de98-4e7b-b86e-1af68be8f6a7","Type":"ContainerStarted","Data":"0376a282f2291aaba62dee631dc543d68c753e1a809f0a51c79a9f95bf7c6da4"}
Jul 31 23:47:17.933559 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:17.933519    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zmrgg" podStartSLOduration=34.494282673 podStartE2EDuration="38.933484906s" totalImagesPullingTime="4.439202233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:17.932323288 +0000 UTC m=+72.805966899" watchObservedRunningTime="2026-07-31 23:47:17.933484906 +0000 UTC m=+72.807128516"
Jul 31 23:47:17.948510 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:17.948463    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sjj4w" podStartSLOduration=66.911096729 podStartE2EDuration="1m11.948445157s" totalImagesPullingTime="5.037348428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:17.947517534 +0000 UTC m=+72.821161145" watchObservedRunningTime="2026-07-31 23:47:17.948445157 +0000 UTC m=+72.822088768"
Jul 31 23:47:20.094493 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:20.094454    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-wpznf\" (UID: \"65c4d1ec-7577-4e31-bc3e-cfed4f74a062\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"
Jul 31 23:47:20.097086 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:20.097056    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/65c4d1ec-7577-4e31-bc3e-cfed4f74a062-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-wpznf\" (UID: \"65c4d1ec-7577-4e31-bc3e-cfed4f74a062\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"
Jul 31 23:47:20.330517 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:20.330478    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6mfmh\""
Jul 31 23:47:20.338808 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:20.338781    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"
Jul 31 23:47:20.852901 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:20.852871    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-wpznf"]
Jul 31 23:47:20.855608 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:47:20.855582    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c4d1ec_7577_4e31_bc3e_cfed4f74a062.slice/crio-7fa533a1edd5f8ffd4a466ce7fbed7ef0c1af69aaa4de22c74a599bdf7a63f05 WatchSource:0}: Error finding container 7fa533a1edd5f8ffd4a466ce7fbed7ef0c1af69aaa4de22c74a599bdf7a63f05: Status 404 returned error can't find the container with id 7fa533a1edd5f8ffd4a466ce7fbed7ef0c1af69aaa4de22c74a599bdf7a63f05
Jul 31 23:47:20.927832 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:20.927805    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf" event={"ID":"65c4d1ec-7577-4e31-bc3e-cfed4f74a062","Type":"ContainerStarted","Data":"7fa533a1edd5f8ffd4a466ce7fbed7ef0c1af69aaa4de22c74a599bdf7a63f05"}
Jul 31 23:47:21.934486 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:21.934450    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w" event={"ID":"19dfca1c-cd68-4742-a4cc-ee7ce492ff30","Type":"ContainerStarted","Data":"85975ea4094e7b08be3790c8c3e1682fcafc72217a0656ffe3f97d99541919fc"}
Jul 31 23:47:21.934845 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:21.934499    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w" event={"ID":"19dfca1c-cd68-4742-a4cc-ee7ce492ff30","Type":"ContainerStarted","Data":"9a85a2ce55b6ee8d471b50bad95a1786b2368f2119e425cf7092cc5ca7868350"}
Jul 31 23:47:21.934845 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:21.934517    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w" event={"ID":"19dfca1c-cd68-4742-a4cc-ee7ce492ff30","Type":"ContainerStarted","Data":"fbc502b887ee34d13e640cad5ac29422e9717d8821a53baf6643c5479d6b195e"}
Jul 31 23:47:21.955499 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:21.955458    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664b6bf69b-dkf5w" podStartSLOduration=2.729142029 podStartE2EDuration="6.955444453s" totalImagesPullingTime="4.226302424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:21.954087275 +0000 UTC m=+76.827730927" watchObservedRunningTime="2026-07-31 23:47:21.955444453 +0000 UTC m=+76.829088041"
Jul 31 23:47:22.939647 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:22.939616    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf" event={"ID":"65c4d1ec-7577-4e31-bc3e-cfed4f74a062","Type":"ContainerStarted","Data":"01b81609ce4388c21efc6350f28eb868959227cdbb03e66429ffaa54ae42ce69"}
Jul 31 23:47:22.957451 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:22.957409    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-wpznf" podStartSLOduration=33.616268291 podStartE2EDuration="34.957393569s" totalImagesPullingTime="1.341125278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:22.955808052 +0000 UTC m=+77.829451663" watchObservedRunningTime="2026-07-31 23:47:22.957393569 +0000 UTC m=+77.831037246"
Jul 31 23:47:27.922283 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:27.922166    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zmrgg"
Jul 31 23:47:34.304689 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.304653    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9pkhd"]
Jul 31 23:47:34.308180 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.308159    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.312284 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.312266    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 23:47:34.312416 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.312397    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 23:47:34.313510 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.313489    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 23:47:34.313612 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.313500    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7cndp\""
Jul 31 23:47:34.313696 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.313679    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 23:47:34.313827 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.313811    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 23:47:34.313974 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.313951    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 23:47:34.396120 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.396095    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvc48\" (UniqueName: \"kubernetes.io/projected/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-kube-api-access-rvc48\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.396231 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.396134    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-node-exporter-textfile\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.396231 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.396160    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-sys\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.396231 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.396183    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-metrics-client-ca\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.396231 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.396207    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-node-exporter-accelerators-collector-config\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.396425 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.396249    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.396425 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.396307    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-root\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.396425 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.396326    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-node-exporter-tls\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.396425 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.396347    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-node-exporter-wtmp\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.497511 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.497488    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-sys\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.497649 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.497517    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-metrics-client-ca\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.497649 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.497546    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-node-exporter-accelerators-collector-config\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.497649 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.497577    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.497649 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.497597    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-sys\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.497649 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.497613    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-root\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.497649 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.497643    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-node-exporter-tls\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.497939 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.497656    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-root\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.497939 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.497669    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-node-exporter-wtmp\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.497939 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.497710    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvc48\" (UniqueName: \"kubernetes.io/projected/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-kube-api-access-rvc48\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.497939 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.497737    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-node-exporter-textfile\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.497939 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.497849    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-node-exporter-wtmp\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.498218 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.498101    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-node-exporter-textfile\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.498218 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.498122    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-metrics-client-ca\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.498357 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.498234    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-node-exporter-accelerators-collector-config\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.499936 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.499917    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.500065 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.500047    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-node-exporter-tls\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.505956 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.505939    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvc48\" (UniqueName: \"kubernetes.io/projected/0283c9e2-278c-41a6-b9ca-3a40d2f5be7c-kube-api-access-rvc48\") pod \"node-exporter-9pkhd\" (UID: \"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c\") " pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.617237 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.617174    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9pkhd"
Jul 31 23:47:34.627395 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:47:34.627362    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0283c9e2_278c_41a6_b9ca_3a40d2f5be7c.slice/crio-6d57f294c01f1bfae928c7a6d3507fe32f68dfbca58d205faf3ee5ee7fd5b8c0 WatchSource:0}: Error finding container 6d57f294c01f1bfae928c7a6d3507fe32f68dfbca58d205faf3ee5ee7fd5b8c0: Status 404 returned error can't find the container with id 6d57f294c01f1bfae928c7a6d3507fe32f68dfbca58d205faf3ee5ee7fd5b8c0
Jul 31 23:47:34.899657 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.899586    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:47:34.987029 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:34.986992    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pkhd" event={"ID":"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c","Type":"ContainerStarted","Data":"6d57f294c01f1bfae928c7a6d3507fe32f68dfbca58d205faf3ee5ee7fd5b8c0"}
Jul 31 23:47:35.992555 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:35.992514    2584 generic.go:356] "Generic (PLEG): container finished" podID="0283c9e2-278c-41a6-b9ca-3a40d2f5be7c" containerID="6a9ceba42c8c7269025e94ea82dfa46415d8d56a8176522cba84c4f722ad94a3" exitCode=0
Jul 31 23:47:35.992956 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:35.992603    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pkhd" event={"ID":"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c","Type":"ContainerDied","Data":"6a9ceba42c8c7269025e94ea82dfa46415d8d56a8176522cba84c4f722ad94a3"}
Jul 31 23:47:36.997941 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:36.997905    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pkhd" event={"ID":"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c","Type":"ContainerStarted","Data":"b3e6edeef6b323df79a8de10a67b8fca37da241cf196114dbc7f4076dd011f58"}
Jul 31 23:47:36.998330 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:36.997989    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pkhd" event={"ID":"0283c9e2-278c-41a6-b9ca-3a40d2f5be7c","Type":"ContainerStarted","Data":"4129fe25fbcbfb3e0239d0c5916c1ce0de871564e479a139af79cf6499ea6a02"}
Jul 31 23:47:37.018288 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:37.018236    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9pkhd" podStartSLOduration=2.170873079 podStartE2EDuration="3.018219764s" totalImagesPullingTime="785.292392ms" totalInitContainerRuntime="62.054293ms" isStatefulPod=true podCreationTimestamp="2026-07-31 23:47:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:37.016491754 +0000 UTC m=+91.890135393" watchObservedRunningTime="2026-07-31 23:47:37.018219764 +0000 UTC m=+91.891863374"
Jul 31 23:47:37.526845 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:37.526814    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"]
Jul 31 23:47:39.493659 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.493623    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"]
Jul 31 23:47:39.496764 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.496748    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.500297 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.500222    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 31 23:47:39.500419 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.500355    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 31 23:47:39.500743 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.500703    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fhp56\""
Jul 31 23:47:39.501073 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.501055    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 31 23:47:39.501566 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.501547    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 31 23:47:39.501899 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.501663    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 31 23:47:39.507444 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.507288    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 31 23:47:39.509562 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.509541    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"]
Jul 31 23:47:39.637659 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.637627    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fc0a74a-6417-4658-9992-85c374fc5649-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.637814 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.637662    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1fc0a74a-6417-4658-9992-85c374fc5649-telemeter-client-tls\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.637814 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.637687    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fc0a74a-6417-4658-9992-85c374fc5649-serving-certs-ca-bundle\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.637814 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.637767    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ls25\" (UniqueName: \"kubernetes.io/projected/1fc0a74a-6417-4658-9992-85c374fc5649-kube-api-access-2ls25\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.637814 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.637800    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1fc0a74a-6417-4658-9992-85c374fc5649-secret-telemeter-client\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.637959 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.637819    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fc0a74a-6417-4658-9992-85c374fc5649-metrics-client-ca\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.637959 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.637893    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fc0a74a-6417-4658-9992-85c374fc5649-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.637959 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.637922    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1fc0a74a-6417-4658-9992-85c374fc5649-federate-client-tls\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.738528 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.738495    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1fc0a74a-6417-4658-9992-85c374fc5649-federate-client-tls\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.738776 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.738536    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fc0a74a-6417-4658-9992-85c374fc5649-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.738776 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.738558    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1fc0a74a-6417-4658-9992-85c374fc5649-telemeter-client-tls\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.738776 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.738578    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fc0a74a-6417-4658-9992-85c374fc5649-serving-certs-ca-bundle\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.738776 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.738606    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ls25\" (UniqueName: \"kubernetes.io/projected/1fc0a74a-6417-4658-9992-85c374fc5649-kube-api-access-2ls25\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.738776 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.738627    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1fc0a74a-6417-4658-9992-85c374fc5649-secret-telemeter-client\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.738776 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.738645    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fc0a74a-6417-4658-9992-85c374fc5649-metrics-client-ca\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.738776 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.738679    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fc0a74a-6417-4658-9992-85c374fc5649-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.739435 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.739404    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fc0a74a-6417-4658-9992-85c374fc5649-serving-certs-ca-bundle\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.739577 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.739491    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fc0a74a-6417-4658-9992-85c374fc5649-metrics-client-ca\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.739702 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.739675    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fc0a74a-6417-4658-9992-85c374fc5649-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.741277 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.741247    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1fc0a74a-6417-4658-9992-85c374fc5649-secret-telemeter-client\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.741386 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.741273    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fc0a74a-6417-4658-9992-85c374fc5649-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.741386 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.741327    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1fc0a74a-6417-4658-9992-85c374fc5649-federate-client-tls\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.741386 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.741336    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1fc0a74a-6417-4658-9992-85c374fc5649-telemeter-client-tls\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.746706 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.746654    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ls25\" (UniqueName: \"kubernetes.io/projected/1fc0a74a-6417-4658-9992-85c374fc5649-kube-api-access-2ls25\") pod \"telemeter-client-75f9cb8c65-g2cxd\" (UID: \"1fc0a74a-6417-4658-9992-85c374fc5649\") " pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.810209 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.810179    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"
Jul 31 23:47:39.947636 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:39.947611    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd"]
Jul 31 23:47:39.953047 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:47:39.953014    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc0a74a_6417_4658_9992_85c374fc5649.slice/crio-9eb2f7ba1de4b4f39b91a93098208ea28aaa5df8ae0a4de9c30b06c83d1a7ca3 WatchSource:0}: Error finding container 9eb2f7ba1de4b4f39b91a93098208ea28aaa5df8ae0a4de9c30b06c83d1a7ca3: Status 404 returned error can't find the container with id 9eb2f7ba1de4b4f39b91a93098208ea28aaa5df8ae0a4de9c30b06c83d1a7ca3
Jul 31 23:47:40.009406 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:40.009344    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd" event={"ID":"1fc0a74a-6417-4658-9992-85c374fc5649","Type":"ContainerStarted","Data":"9eb2f7ba1de4b4f39b91a93098208ea28aaa5df8ae0a4de9c30b06c83d1a7ca3"}
Jul 31 23:47:42.018549 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:42.018509    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd" event={"ID":"1fc0a74a-6417-4658-9992-85c374fc5649","Type":"ContainerStarted","Data":"6f33c3cc286e58370fcda977bbf87a986b2f5e96312f875eb3749a1299107d56"}
Jul 31 23:47:43.024402 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:43.024368    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd" event={"ID":"1fc0a74a-6417-4658-9992-85c374fc5649","Type":"ContainerStarted","Data":"229663336db19c5280160c5140e7f45505011f49c1e248d4caaeabc34bb5372c"}
Jul 31 23:47:43.024761 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:43.024426    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd" event={"ID":"1fc0a74a-6417-4658-9992-85c374fc5649","Type":"ContainerStarted","Data":"54d704e4d7267bd9eceecaaa11ba555aa6854bdfd302bf84f434dc870d79ede6"}
Jul 31 23:47:43.055450 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:43.055398    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-75f9cb8c65-g2cxd" podStartSLOduration=1.663331322 podStartE2EDuration="4.055378202s" totalImagesPullingTime="2.39204688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:43.052914686 +0000 UTC m=+97.926558307" watchObservedRunningTime="2026-07-31 23:47:43.055378202 +0000 UTC m=+97.929021813"
Jul 31 23:47:47.917101 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:47:47.917071    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qpmwv"
Jul 31 23:48:02.546137 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.546079    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x" podUID="3ea4e9da-91c8-4410-969d-49f53895860e" containerName="registry" containerID="cri-o://50f38e164e144fceaba3828cddf7f9d89b25f865b732299f5c633fdb876e8287" gracePeriod=30
Jul 31 23:48:02.785853 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.785825    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:48:02.808789 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.808726    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls\") pod \"3ea4e9da-91c8-4410-969d-49f53895860e\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") "
Jul 31 23:48:02.808789 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.808770    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-bound-sa-token\") pod \"3ea4e9da-91c8-4410-969d-49f53895860e\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") "
Jul 31 23:48:02.808954 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.808814    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ea4e9da-91c8-4410-969d-49f53895860e-registry-certificates\") pod \"3ea4e9da-91c8-4410-969d-49f53895860e\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") "
Jul 31 23:48:02.808954 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.808843    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ea4e9da-91c8-4410-969d-49f53895860e-installation-pull-secrets\") pod \"3ea4e9da-91c8-4410-969d-49f53895860e\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") "
Jul 31 23:48:02.808954 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.808867    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ea4e9da-91c8-4410-969d-49f53895860e-image-registry-private-configuration\") pod \"3ea4e9da-91c8-4410-969d-49f53895860e\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") "
Jul 31 23:48:02.808954 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.808901    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-572ss\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-kube-api-access-572ss\") pod \"3ea4e9da-91c8-4410-969d-49f53895860e\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") "
Jul 31 23:48:02.808954 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.808926    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ea4e9da-91c8-4410-969d-49f53895860e-trusted-ca\") pod \"3ea4e9da-91c8-4410-969d-49f53895860e\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") "
Jul 31 23:48:02.809209 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.808988    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ea4e9da-91c8-4410-969d-49f53895860e-ca-trust-extracted\") pod \"3ea4e9da-91c8-4410-969d-49f53895860e\" (UID: \"3ea4e9da-91c8-4410-969d-49f53895860e\") "
Jul 31 23:48:02.809280 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.809248    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea4e9da-91c8-4410-969d-49f53895860e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3ea4e9da-91c8-4410-969d-49f53895860e" (UID: "3ea4e9da-91c8-4410-969d-49f53895860e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:02.810468 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.810442    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea4e9da-91c8-4410-969d-49f53895860e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3ea4e9da-91c8-4410-969d-49f53895860e" (UID: "3ea4e9da-91c8-4410-969d-49f53895860e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:02.811870 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.811802    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea4e9da-91c8-4410-969d-49f53895860e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3ea4e9da-91c8-4410-969d-49f53895860e" (UID: "3ea4e9da-91c8-4410-969d-49f53895860e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:02.812020 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.811924    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3ea4e9da-91c8-4410-969d-49f53895860e" (UID: "3ea4e9da-91c8-4410-969d-49f53895860e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:48:02.812020 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.812003    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3ea4e9da-91c8-4410-969d-49f53895860e" (UID: "3ea4e9da-91c8-4410-969d-49f53895860e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:48:02.812213 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.812184    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-kube-api-access-572ss" (OuterVolumeSpecName: "kube-api-access-572ss") pod "3ea4e9da-91c8-4410-969d-49f53895860e" (UID: "3ea4e9da-91c8-4410-969d-49f53895860e"). InnerVolumeSpecName "kube-api-access-572ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:48:02.812350 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.812311    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea4e9da-91c8-4410-969d-49f53895860e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3ea4e9da-91c8-4410-969d-49f53895860e" (UID: "3ea4e9da-91c8-4410-969d-49f53895860e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:02.820790 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.820766    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea4e9da-91c8-4410-969d-49f53895860e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3ea4e9da-91c8-4410-969d-49f53895860e" (UID: "3ea4e9da-91c8-4410-969d-49f53895860e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:48:02.909803 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.909771    2584 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-bound-sa-token\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:48:02.909803 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.909798    2584 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ea4e9da-91c8-4410-969d-49f53895860e-registry-certificates\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:48:02.909803 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.909807    2584 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ea4e9da-91c8-4410-969d-49f53895860e-installation-pull-secrets\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:48:02.910031 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.909818    2584 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ea4e9da-91c8-4410-969d-49f53895860e-image-registry-private-configuration\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:48:02.910031 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.909827    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-572ss\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-kube-api-access-572ss\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:48:02.910031 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.909835    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ea4e9da-91c8-4410-969d-49f53895860e-trusted-ca\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:48:02.910031 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.909844    2584 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ea4e9da-91c8-4410-969d-49f53895860e-ca-trust-extracted\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:48:02.910031 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:02.909852    2584 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ea4e9da-91c8-4410-969d-49f53895860e-registry-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:48:03.093724 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:03.093638    2584 generic.go:356] "Generic (PLEG): container finished" podID="3ea4e9da-91c8-4410-969d-49f53895860e" containerID="50f38e164e144fceaba3828cddf7f9d89b25f865b732299f5c633fdb876e8287" exitCode=0
Jul 31 23:48:03.093724 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:03.093686    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"
Jul 31 23:48:03.093724 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:03.093699    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x" event={"ID":"3ea4e9da-91c8-4410-969d-49f53895860e","Type":"ContainerDied","Data":"50f38e164e144fceaba3828cddf7f9d89b25f865b732299f5c633fdb876e8287"}
Jul 31 23:48:03.094013 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:03.093780    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x" event={"ID":"3ea4e9da-91c8-4410-969d-49f53895860e","Type":"ContainerDied","Data":"edee64e17ab6fbab28116db3735e274a209c31ca6bdb87044af8ccab6a5cb088"}
Jul 31 23:48:03.094013 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:03.093807    2584 scope.go:117] "RemoveContainer" containerID="50f38e164e144fceaba3828cddf7f9d89b25f865b732299f5c633fdb876e8287"
Jul 31 23:48:03.102132 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:03.102115    2584 scope.go:117] "RemoveContainer" containerID="50f38e164e144fceaba3828cddf7f9d89b25f865b732299f5c633fdb876e8287"
Jul 31 23:48:03.102392 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:48:03.102359    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f38e164e144fceaba3828cddf7f9d89b25f865b732299f5c633fdb876e8287\": container with ID starting with 50f38e164e144fceaba3828cddf7f9d89b25f865b732299f5c633fdb876e8287 not found: ID does not exist" containerID="50f38e164e144fceaba3828cddf7f9d89b25f865b732299f5c633fdb876e8287"
Jul 31 23:48:03.102433 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:03.102399    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f38e164e144fceaba3828cddf7f9d89b25f865b732299f5c633fdb876e8287"} err="failed to get container status \"50f38e164e144fceaba3828cddf7f9d89b25f865b732299f5c633fdb876e8287\": rpc error: code = NotFound desc = could not find container \"50f38e164e144fceaba3828cddf7f9d89b25f865b732299f5c633fdb876e8287\": container with ID starting with 50f38e164e144fceaba3828cddf7f9d89b25f865b732299f5c633fdb876e8287 not found: ID does not exist"
Jul 31 23:48:03.116546 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:03.116520    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"]
Jul 31 23:48:03.122484 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:03.122464    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5cfbfcfdcd-tlc8x"]
Jul 31 23:48:03.570391 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:03.570363    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ea4e9da-91c8-4410-969d-49f53895860e" path="/var/lib/kubelet/pods/3ea4e9da-91c8-4410-969d-49f53895860e/volumes"
Jul 31 23:48:23.165002 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:23.164954    2584 generic.go:356] "Generic (PLEG): container finished" podID="45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b" containerID="cc92992690bbc15d380af5d942cdf4441bfa7914e75ccb36f560221112cc0a7b" exitCode=0
Jul 31 23:48:23.165406 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:23.165027    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-rp75j" event={"ID":"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b","Type":"ContainerDied","Data":"cc92992690bbc15d380af5d942cdf4441bfa7914e75ccb36f560221112cc0a7b"}
Jul 31 23:48:23.165406 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:23.165336    2584 scope.go:117] "RemoveContainer" containerID="cc92992690bbc15d380af5d942cdf4441bfa7914e75ccb36f560221112cc0a7b"
Jul 31 23:48:24.169944 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:24.169910    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-rp75j" event={"ID":"45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b","Type":"ContainerStarted","Data":"d1ee88b9d2539bfdfb709424b835fdf6a6deca228507144325b3f6428bc9529f"}
Jul 31 23:48:28.184483 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:28.184451    2584 generic.go:356] "Generic (PLEG): container finished" podID="51994692-c2a6-4b7e-a1c3-ca8255d407e8" containerID="e6a7ba6826d4da93ef9c843d7401795bff31b1252df926e7d3b2fe08036e5600" exitCode=0
Jul 31 23:48:28.184929 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:28.184497    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97" event={"ID":"51994692-c2a6-4b7e-a1c3-ca8255d407e8","Type":"ContainerDied","Data":"e6a7ba6826d4da93ef9c843d7401795bff31b1252df926e7d3b2fe08036e5600"}
Jul 31 23:48:28.184929 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:28.184836    2584 scope.go:117] "RemoveContainer" containerID="e6a7ba6826d4da93ef9c843d7401795bff31b1252df926e7d3b2fe08036e5600"
Jul 31 23:48:29.193459 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:48:29.193422    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nmn97" event={"ID":"51994692-c2a6-4b7e-a1c3-ca8255d407e8","Type":"ContainerStarted","Data":"59d99058e16683131b6d8eb0a6cac0ee4edd00a39b07cadbb21311661e853e2d"}
Jul 31 23:50:23.089131 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.089095    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"]
Jul 31 23:50:23.089601 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.089359    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea4e9da-91c8-4410-969d-49f53895860e" containerName="registry"
Jul 31 23:50:23.089601 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.089373    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea4e9da-91c8-4410-969d-49f53895860e" containerName="registry"
Jul 31 23:50:23.089601 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.089426    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ea4e9da-91c8-4410-969d-49f53895860e" containerName="registry"
Jul 31 23:50:23.092313 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.092295    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"
Jul 31 23:50:23.094880 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.094851    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 31 23:50:23.095037 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.094920    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jxjvw\""
Jul 31 23:50:23.095863 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.095846    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 31 23:50:23.101609 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.101590    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"]
Jul 31 23:50:23.207796 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.207769    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnn8h\" (UniqueName: \"kubernetes.io/projected/0a9bdf09-059d-49ee-94af-29499d7daf4e-kube-api-access-nnn8h\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz\" (UID: \"0a9bdf09-059d-49ee-94af-29499d7daf4e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"
Jul 31 23:50:23.207929 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.207799    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a9bdf09-059d-49ee-94af-29499d7daf4e-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz\" (UID: \"0a9bdf09-059d-49ee-94af-29499d7daf4e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"
Jul 31 23:50:23.207929 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.207834    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a9bdf09-059d-49ee-94af-29499d7daf4e-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz\" (UID: \"0a9bdf09-059d-49ee-94af-29499d7daf4e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"
Jul 31 23:50:23.308274 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.308249    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a9bdf09-059d-49ee-94af-29499d7daf4e-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz\" (UID: \"0a9bdf09-059d-49ee-94af-29499d7daf4e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"
Jul 31 23:50:23.308380 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.308297    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnn8h\" (UniqueName: \"kubernetes.io/projected/0a9bdf09-059d-49ee-94af-29499d7daf4e-kube-api-access-nnn8h\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz\" (UID: \"0a9bdf09-059d-49ee-94af-29499d7daf4e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"
Jul 31 23:50:23.308380 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.308319    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a9bdf09-059d-49ee-94af-29499d7daf4e-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz\" (UID: \"0a9bdf09-059d-49ee-94af-29499d7daf4e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"
Jul 31 23:50:23.308610 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.308593    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a9bdf09-059d-49ee-94af-29499d7daf4e-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz\" (UID: \"0a9bdf09-059d-49ee-94af-29499d7daf4e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"
Jul 31 23:50:23.308645 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.308633    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a9bdf09-059d-49ee-94af-29499d7daf4e-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz\" (UID: \"0a9bdf09-059d-49ee-94af-29499d7daf4e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"
Jul 31 23:50:23.316462 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.316444    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnn8h\" (UniqueName: \"kubernetes.io/projected/0a9bdf09-059d-49ee-94af-29499d7daf4e-kube-api-access-nnn8h\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz\" (UID: \"0a9bdf09-059d-49ee-94af-29499d7daf4e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"
Jul 31 23:50:23.400779 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.400727    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"
Jul 31 23:50:23.526578 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.526413    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"]
Jul 31 23:50:23.529213 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:50:23.529185    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9bdf09_059d_49ee_94af_29499d7daf4e.slice/crio-21a0249b3197716d60663f4b25a4c873c4fce16a24e5f0eec227106f021b7d96 WatchSource:0}: Error finding container 21a0249b3197716d60663f4b25a4c873c4fce16a24e5f0eec227106f021b7d96: Status 404 returned error can't find the container with id 21a0249b3197716d60663f4b25a4c873c4fce16a24e5f0eec227106f021b7d96
Jul 31 23:50:23.597070 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:23.597045    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz" event={"ID":"0a9bdf09-059d-49ee-94af-29499d7daf4e","Type":"ContainerStarted","Data":"21a0249b3197716d60663f4b25a4c873c4fce16a24e5f0eec227106f021b7d96"}
Jul 31 23:50:30.628879 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:30.628846    2584 generic.go:356] "Generic (PLEG): container finished" podID="0a9bdf09-059d-49ee-94af-29499d7daf4e" containerID="5a45d1fbd2e4e95b30e5c76f14b87afa0dde2f1fbb418e23d696d8591501982c" exitCode=0
Jul 31 23:50:30.629275 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:30.628944    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz" event={"ID":"0a9bdf09-059d-49ee-94af-29499d7daf4e","Type":"ContainerDied","Data":"5a45d1fbd2e4e95b30e5c76f14b87afa0dde2f1fbb418e23d696d8591501982c"}
Jul 31 23:50:33.641350 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:33.641313    2584 generic.go:356] "Generic (PLEG): container finished" podID="0a9bdf09-059d-49ee-94af-29499d7daf4e" containerID="ea6b3a44c28900377b85c85668b7af4cb0de6e3611bbbefadf18d2b2a14d1a59" exitCode=0
Jul 31 23:50:33.641810 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:33.641376    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz" event={"ID":"0a9bdf09-059d-49ee-94af-29499d7daf4e","Type":"ContainerDied","Data":"ea6b3a44c28900377b85c85668b7af4cb0de6e3611bbbefadf18d2b2a14d1a59"}
Jul 31 23:50:40.670856 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:40.670819    2584 generic.go:356] "Generic (PLEG): container finished" podID="0a9bdf09-059d-49ee-94af-29499d7daf4e" containerID="cb9239f13f68d765e0284a422c8d77592fd1f11ddc896de8024dfae716712ce1" exitCode=0
Jul 31 23:50:40.671265 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:40.670881    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz" event={"ID":"0a9bdf09-059d-49ee-94af-29499d7daf4e","Type":"ContainerDied","Data":"cb9239f13f68d765e0284a422c8d77592fd1f11ddc896de8024dfae716712ce1"}
Jul 31 23:50:41.793710 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:41.793688    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"
Jul 31 23:50:41.959737 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:41.959673    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a9bdf09-059d-49ee-94af-29499d7daf4e-bundle\") pod \"0a9bdf09-059d-49ee-94af-29499d7daf4e\" (UID: \"0a9bdf09-059d-49ee-94af-29499d7daf4e\") "
Jul 31 23:50:41.959737 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:41.959726    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a9bdf09-059d-49ee-94af-29499d7daf4e-util\") pod \"0a9bdf09-059d-49ee-94af-29499d7daf4e\" (UID: \"0a9bdf09-059d-49ee-94af-29499d7daf4e\") "
Jul 31 23:50:41.959886 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:41.959746    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnn8h\" (UniqueName: \"kubernetes.io/projected/0a9bdf09-059d-49ee-94af-29499d7daf4e-kube-api-access-nnn8h\") pod \"0a9bdf09-059d-49ee-94af-29499d7daf4e\" (UID: \"0a9bdf09-059d-49ee-94af-29499d7daf4e\") "
Jul 31 23:50:41.960355 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:41.960322    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9bdf09-059d-49ee-94af-29499d7daf4e-bundle" (OuterVolumeSpecName: "bundle") pod "0a9bdf09-059d-49ee-94af-29499d7daf4e" (UID: "0a9bdf09-059d-49ee-94af-29499d7daf4e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:50:41.961828 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:41.961805    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9bdf09-059d-49ee-94af-29499d7daf4e-kube-api-access-nnn8h" (OuterVolumeSpecName: "kube-api-access-nnn8h") pod "0a9bdf09-059d-49ee-94af-29499d7daf4e" (UID: "0a9bdf09-059d-49ee-94af-29499d7daf4e"). InnerVolumeSpecName "kube-api-access-nnn8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:50:41.965868 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:41.965841    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9bdf09-059d-49ee-94af-29499d7daf4e-util" (OuterVolumeSpecName: "util") pod "0a9bdf09-059d-49ee-94af-29499d7daf4e" (UID: "0a9bdf09-059d-49ee-94af-29499d7daf4e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:50:42.060855 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:42.060819    2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a9bdf09-059d-49ee-94af-29499d7daf4e-util\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:50:42.060855 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:42.060848    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnn8h\" (UniqueName: \"kubernetes.io/projected/0a9bdf09-059d-49ee-94af-29499d7daf4e-kube-api-access-nnn8h\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:50:42.060855 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:42.060859    2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a9bdf09-059d-49ee-94af-29499d7daf4e-bundle\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:50:42.679351 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:42.679311    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz" event={"ID":"0a9bdf09-059d-49ee-94af-29499d7daf4e","Type":"ContainerDied","Data":"21a0249b3197716d60663f4b25a4c873c4fce16a24e5f0eec227106f021b7d96"}
Jul 31 23:50:42.679351 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:42.679353    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b596qwz"
Jul 31 23:50:42.679546 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:42.679361    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21a0249b3197716d60663f4b25a4c873c4fce16a24e5f0eec227106f021b7d96"
Jul 31 23:50:45.492022 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.491985    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg"]
Jul 31 23:50:45.492421 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.492257    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9bdf09-059d-49ee-94af-29499d7daf4e" containerName="pull"
Jul 31 23:50:45.492421 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.492268    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9bdf09-059d-49ee-94af-29499d7daf4e" containerName="pull"
Jul 31 23:50:45.492421 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.492277    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9bdf09-059d-49ee-94af-29499d7daf4e" containerName="extract"
Jul 31 23:50:45.492421 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.492282    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9bdf09-059d-49ee-94af-29499d7daf4e" containerName="extract"
Jul 31 23:50:45.492421 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.492293    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9bdf09-059d-49ee-94af-29499d7daf4e" containerName="util"
Jul 31 23:50:45.492421 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.492300    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9bdf09-059d-49ee-94af-29499d7daf4e" containerName="util"
Jul 31 23:50:45.492421 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.492339    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a9bdf09-059d-49ee-94af-29499d7daf4e" containerName="extract"
Jul 31 23:50:45.494179 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.494164    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg"
Jul 31 23:50:45.497648 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.497624    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-gw4r7\""
Jul 31 23:50:45.497775 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.497695    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 31 23:50:45.497838 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.497824    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 31 23:50:45.498702 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.498687    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 31 23:50:45.509168 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.509148    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg"]
Jul 31 23:50:45.587222 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.587194    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ee3b1ee8-3f03-4685-8138-8363ca0fc16b-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg\" (UID: \"ee3b1ee8-3f03-4685-8138-8363ca0fc16b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg"
Jul 31 23:50:45.587383 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.587241    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q22b7\" (UniqueName: \"kubernetes.io/projected/ee3b1ee8-3f03-4685-8138-8363ca0fc16b-kube-api-access-q22b7\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg\" (UID: \"ee3b1ee8-3f03-4685-8138-8363ca0fc16b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg"
Jul 31 23:50:45.687796 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.687761    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ee3b1ee8-3f03-4685-8138-8363ca0fc16b-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg\" (UID: \"ee3b1ee8-3f03-4685-8138-8363ca0fc16b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg"
Jul 31 23:50:45.687996 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.687839    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q22b7\" (UniqueName: \"kubernetes.io/projected/ee3b1ee8-3f03-4685-8138-8363ca0fc16b-kube-api-access-q22b7\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg\" (UID: \"ee3b1ee8-3f03-4685-8138-8363ca0fc16b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg"
Jul 31 23:50:45.690368 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.690344    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ee3b1ee8-3f03-4685-8138-8363ca0fc16b-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg\" (UID: \"ee3b1ee8-3f03-4685-8138-8363ca0fc16b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg"
Jul 31 23:50:45.696800 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.696776    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q22b7\" (UniqueName: \"kubernetes.io/projected/ee3b1ee8-3f03-4685-8138-8363ca0fc16b-kube-api-access-q22b7\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg\" (UID: \"ee3b1ee8-3f03-4685-8138-8363ca0fc16b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg"
Jul 31 23:50:45.803834 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.803809    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg"
Jul 31 23:50:45.953619 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:45.953597    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg"]
Jul 31 23:50:45.955564 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:50:45.955533    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3b1ee8_3f03_4685_8138_8363ca0fc16b.slice/crio-7c303d9b7c7154165a2e5610f6986606989837b8da6d28f27447cc0e2bea669f WatchSource:0}: Error finding container 7c303d9b7c7154165a2e5610f6986606989837b8da6d28f27447cc0e2bea669f: Status 404 returned error can't find the container with id 7c303d9b7c7154165a2e5610f6986606989837b8da6d28f27447cc0e2bea669f
Jul 31 23:50:46.695847 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:46.695798    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg" event={"ID":"ee3b1ee8-3f03-4685-8138-8363ca0fc16b","Type":"ContainerStarted","Data":"7c303d9b7c7154165a2e5610f6986606989837b8da6d28f27447cc0e2bea669f"}
Jul 31 23:50:48.705051 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:48.704938    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg" event={"ID":"ee3b1ee8-3f03-4685-8138-8363ca0fc16b","Type":"ContainerStarted","Data":"fc1d20fea2064a784649c8c3d256dc30ffea03d60c72651cc2c79ed228e35d75"}
Jul 31 23:50:48.705418 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:48.705072    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg"
Jul 31 23:50:48.729584 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:48.729533    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg" podStartSLOduration=1.25897017 podStartE2EDuration="3.72951944s" totalImagesPullingTime="2.47054927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:50:48.728535149 +0000 UTC m=+283.602178762" watchObservedRunningTime="2026-07-31 23:50:48.72951944 +0000 UTC m=+283.603163050"
Jul 31 23:50:49.207009 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.206975    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-8ztzw"]
Jul 31 23:50:49.209000 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.208986    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-8ztzw"
Jul 31 23:50:49.212075 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.212054    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 31 23:50:49.212203 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.212116    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-sbch5\""
Jul 31 23:50:49.212203 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.212151    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 31 23:50:49.220321 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.220301    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-8ztzw"]
Jul 31 23:50:49.321361 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.321325    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9be9a280-442e-4821-a566-b0340cefbcdb-cabundle0\") pod \"keda-operator-77f648d694-8ztzw\" (UID: \"9be9a280-442e-4821-a566-b0340cefbcdb\") " pod="openshift-keda/keda-operator-77f648d694-8ztzw"
Jul 31 23:50:49.321519 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.321430    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9be9a280-442e-4821-a566-b0340cefbcdb-certificates\") pod \"keda-operator-77f648d694-8ztzw\" (UID: \"9be9a280-442e-4821-a566-b0340cefbcdb\") " pod="openshift-keda/keda-operator-77f648d694-8ztzw"
Jul 31 23:50:49.321519 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.321462    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4npnp\" (UniqueName: \"kubernetes.io/projected/9be9a280-442e-4821-a566-b0340cefbcdb-kube-api-access-4npnp\") pod \"keda-operator-77f648d694-8ztzw\" (UID: \"9be9a280-442e-4821-a566-b0340cefbcdb\") " pod="openshift-keda/keda-operator-77f648d694-8ztzw"
Jul 31 23:50:49.422043 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.422007    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9be9a280-442e-4821-a566-b0340cefbcdb-cabundle0\") pod \"keda-operator-77f648d694-8ztzw\" (UID: \"9be9a280-442e-4821-a566-b0340cefbcdb\") " pod="openshift-keda/keda-operator-77f648d694-8ztzw"
Jul 31 23:50:49.422217 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.422064    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9be9a280-442e-4821-a566-b0340cefbcdb-certificates\") pod \"keda-operator-77f648d694-8ztzw\" (UID: \"9be9a280-442e-4821-a566-b0340cefbcdb\") " pod="openshift-keda/keda-operator-77f648d694-8ztzw"
Jul 31 23:50:49.422217 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.422095    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4npnp\" (UniqueName: \"kubernetes.io/projected/9be9a280-442e-4821-a566-b0340cefbcdb-kube-api-access-4npnp\") pod \"keda-operator-77f648d694-8ztzw\" (UID: \"9be9a280-442e-4821-a566-b0340cefbcdb\") " pod="openshift-keda/keda-operator-77f648d694-8ztzw"
Jul 31 23:50:49.422324 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:50:49.422227    2584 secret.go:281] references non-existent secret key: ca.crt
Jul 31 23:50:49.422324 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:50:49.422245    2584 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 31 23:50:49.422324 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:50:49.422253    2584 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-8ztzw: references non-existent secret key: ca.crt
Jul 31 23:50:49.422324 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:50:49.422320    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9be9a280-442e-4821-a566-b0340cefbcdb-certificates podName:9be9a280-442e-4821-a566-b0340cefbcdb nodeName:}" failed. No retries permitted until 2026-07-31 23:50:49.922302348 +0000 UTC m=+284.795945940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/9be9a280-442e-4821-a566-b0340cefbcdb-certificates") pod "keda-operator-77f648d694-8ztzw" (UID: "9be9a280-442e-4821-a566-b0340cefbcdb") : references non-existent secret key: ca.crt
Jul 31 23:50:49.422786 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.422766    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9be9a280-442e-4821-a566-b0340cefbcdb-cabundle0\") pod \"keda-operator-77f648d694-8ztzw\" (UID: \"9be9a280-442e-4821-a566-b0340cefbcdb\") " pod="openshift-keda/keda-operator-77f648d694-8ztzw"
Jul 31 23:50:49.443242 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.443219    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4npnp\" (UniqueName: \"kubernetes.io/projected/9be9a280-442e-4821-a566-b0340cefbcdb-kube-api-access-4npnp\") pod \"keda-operator-77f648d694-8ztzw\" (UID: \"9be9a280-442e-4821-a566-b0340cefbcdb\") " pod="openshift-keda/keda-operator-77f648d694-8ztzw"
Jul 31 23:50:49.927667 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.927636    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9be9a280-442e-4821-a566-b0340cefbcdb-certificates\") pod \"keda-operator-77f648d694-8ztzw\" (UID: \"9be9a280-442e-4821-a566-b0340cefbcdb\") " pod="openshift-keda/keda-operator-77f648d694-8ztzw"
Jul 31 23:50:49.930011 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:49.929992    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9be9a280-442e-4821-a566-b0340cefbcdb-certificates\") pod \"keda-operator-77f648d694-8ztzw\" (UID: \"9be9a280-442e-4821-a566-b0340cefbcdb\") " pod="openshift-keda/keda-operator-77f648d694-8ztzw"
Jul 31 23:50:50.119206 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:50.119166    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-8ztzw"
Jul 31 23:50:50.247874 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:50.247852    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-8ztzw"]
Jul 31 23:50:50.249921 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:50:50.249895    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be9a280_442e_4821_a566_b0340cefbcdb.slice/crio-11e594e5839e85116c548f58308dfaef2c7b11e1672114445a3b249bb4dd0b11 WatchSource:0}: Error finding container 11e594e5839e85116c548f58308dfaef2c7b11e1672114445a3b249bb4dd0b11: Status 404 returned error can't find the container with id 11e594e5839e85116c548f58308dfaef2c7b11e1672114445a3b249bb4dd0b11
Jul 31 23:50:50.713749 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:50.713635    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-8ztzw" event={"ID":"9be9a280-442e-4821-a566-b0340cefbcdb","Type":"ContainerStarted","Data":"11e594e5839e85116c548f58308dfaef2c7b11e1672114445a3b249bb4dd0b11"}
Jul 31 23:50:53.726712 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:53.726612    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-8ztzw" event={"ID":"9be9a280-442e-4821-a566-b0340cefbcdb","Type":"ContainerStarted","Data":"67395402877e5ab99e197ac88248478efc98bae6ee34eef041da501fe0b49865"}
Jul 31 23:50:53.727115 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:53.726741    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-8ztzw"
Jul 31 23:50:53.755235 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:50:53.755190    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-8ztzw" podStartSLOduration=1.627816507 podStartE2EDuration="4.755170236s" totalImagesPullingTime="3.127353729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:50:53.754565 +0000 UTC m=+288.628208610" watchObservedRunningTime="2026-07-31 23:50:53.755170236 +0000 UTC m=+288.628813858"
Jul 31 23:51:05.488048 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:51:05.488018    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Jul 31 23:51:05.488730 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:51:05.488712    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Jul 31 23:51:05.499248 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:51:05.499229    2584 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 23:51:09.710692 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:51:09.710664    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7hcrg"
Jul 31 23:51:14.732852 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:51:14.732822    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-8ztzw"
Jul 31 23:52:11.241184 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.241107    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-jpzhl"]
Jul 31 23:52:11.243808 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.243792    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-jpzhl"
Jul 31 23:52:11.246325 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.246303    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 31 23:52:11.246470 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.246336    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 31 23:52:11.246470 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.246399    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 31 23:52:11.247560 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.247537    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-k99sj\""
Jul 31 23:52:11.257238 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.257214    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-jpzhl"]
Jul 31 23:52:11.287464 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.287430    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-2ldmx"]
Jul 31 23:52:11.289555 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.289535    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2ldmx"
Jul 31 23:52:11.292058 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.292039    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 31 23:52:11.292776 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.292758    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-rvpq8\""
Jul 31 23:52:11.301799 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.301778    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2ldmx"]
Jul 31 23:52:11.311733 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.311696    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42q52\" (UniqueName: \"kubernetes.io/projected/13860f72-8141-4c7e-849f-afbfd6911c69-kube-api-access-42q52\") pod \"kserve-controller-manager-674d959fb8-jpzhl\" (UID: \"13860f72-8141-4c7e-849f-afbfd6911c69\") " pod="kserve/kserve-controller-manager-674d959fb8-jpzhl"
Jul 31 23:52:11.311733 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.311735    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13860f72-8141-4c7e-849f-afbfd6911c69-cert\") pod \"kserve-controller-manager-674d959fb8-jpzhl\" (UID: \"13860f72-8141-4c7e-849f-afbfd6911c69\") " pod="kserve/kserve-controller-manager-674d959fb8-jpzhl"
Jul 31 23:52:11.413144 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.413102    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dksq2\" (UniqueName: \"kubernetes.io/projected/ba2f18cd-6d28-4e57-aa42-f8d2f0e46f32-kube-api-access-dksq2\") pod \"seaweedfs-64568bcd49-2ldmx\" (UID: \"ba2f18cd-6d28-4e57-aa42-f8d2f0e46f32\") " pod="kserve/seaweedfs-64568bcd49-2ldmx"
Jul 31 23:52:11.413144 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.413149    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42q52\" (UniqueName: \"kubernetes.io/projected/13860f72-8141-4c7e-849f-afbfd6911c69-kube-api-access-42q52\") pod \"kserve-controller-manager-674d959fb8-jpzhl\" (UID: \"13860f72-8141-4c7e-849f-afbfd6911c69\") " pod="kserve/kserve-controller-manager-674d959fb8-jpzhl"
Jul 31 23:52:11.413343 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.413179    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13860f72-8141-4c7e-849f-afbfd6911c69-cert\") pod \"kserve-controller-manager-674d959fb8-jpzhl\" (UID: \"13860f72-8141-4c7e-849f-afbfd6911c69\") " pod="kserve/kserve-controller-manager-674d959fb8-jpzhl"
Jul 31 23:52:11.413343 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.413218    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ba2f18cd-6d28-4e57-aa42-f8d2f0e46f32-data\") pod \"seaweedfs-64568bcd49-2ldmx\" (UID: \"ba2f18cd-6d28-4e57-aa42-f8d2f0e46f32\") " pod="kserve/seaweedfs-64568bcd49-2ldmx"
Jul 31 23:52:11.413343 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:52:11.413322    2584 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 31 23:52:11.413440 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:52:11.413388    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13860f72-8141-4c7e-849f-afbfd6911c69-cert podName:13860f72-8141-4c7e-849f-afbfd6911c69 nodeName:}" failed. No retries permitted until 2026-07-31 23:52:11.913371289 +0000 UTC m=+366.787014880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/13860f72-8141-4c7e-849f-afbfd6911c69-cert") pod "kserve-controller-manager-674d959fb8-jpzhl" (UID: "13860f72-8141-4c7e-849f-afbfd6911c69") : secret "kserve-webhook-server-cert" not found
Jul 31 23:52:11.424689 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.424655    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42q52\" (UniqueName: \"kubernetes.io/projected/13860f72-8141-4c7e-849f-afbfd6911c69-kube-api-access-42q52\") pod \"kserve-controller-manager-674d959fb8-jpzhl\" (UID: \"13860f72-8141-4c7e-849f-afbfd6911c69\") " pod="kserve/kserve-controller-manager-674d959fb8-jpzhl"
Jul 31 23:52:11.514435 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.514346    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dksq2\" (UniqueName: \"kubernetes.io/projected/ba2f18cd-6d28-4e57-aa42-f8d2f0e46f32-kube-api-access-dksq2\") pod \"seaweedfs-64568bcd49-2ldmx\" (UID: \"ba2f18cd-6d28-4e57-aa42-f8d2f0e46f32\") " pod="kserve/seaweedfs-64568bcd49-2ldmx"
Jul 31 23:52:11.514631 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.514441    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ba2f18cd-6d28-4e57-aa42-f8d2f0e46f32-data\") pod \"seaweedfs-64568bcd49-2ldmx\" (UID: \"ba2f18cd-6d28-4e57-aa42-f8d2f0e46f32\") " pod="kserve/seaweedfs-64568bcd49-2ldmx"
Jul 31 23:52:11.514850 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.514829    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ba2f18cd-6d28-4e57-aa42-f8d2f0e46f32-data\") pod \"seaweedfs-64568bcd49-2ldmx\" (UID: \"ba2f18cd-6d28-4e57-aa42-f8d2f0e46f32\") " pod="kserve/seaweedfs-64568bcd49-2ldmx"
Jul 31 23:52:11.522824 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.522796    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dksq2\" (UniqueName: \"kubernetes.io/projected/ba2f18cd-6d28-4e57-aa42-f8d2f0e46f32-kube-api-access-dksq2\") pod \"seaweedfs-64568bcd49-2ldmx\" (UID: \"ba2f18cd-6d28-4e57-aa42-f8d2f0e46f32\") " pod="kserve/seaweedfs-64568bcd49-2ldmx"
Jul 31 23:52:11.599363 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.599341    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2ldmx"
Jul 31 23:52:11.724932 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.724912    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2ldmx"]
Jul 31 23:52:11.727121 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:52:11.727098    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2f18cd_6d28_4e57_aa42_f8d2f0e46f32.slice/crio-898b1f953ee766e039f4f2128419933b636920e73aed325905736c98f67dcda6 WatchSource:0}: Error finding container 898b1f953ee766e039f4f2128419933b636920e73aed325905736c98f67dcda6: Status 404 returned error can't find the container with id 898b1f953ee766e039f4f2128419933b636920e73aed325905736c98f67dcda6
Jul 31 23:52:11.728414 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.728396    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:52:11.917778 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.917748    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13860f72-8141-4c7e-849f-afbfd6911c69-cert\") pod \"kserve-controller-manager-674d959fb8-jpzhl\" (UID: \"13860f72-8141-4c7e-849f-afbfd6911c69\") " pod="kserve/kserve-controller-manager-674d959fb8-jpzhl"
Jul 31 23:52:11.920041 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:11.920023    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13860f72-8141-4c7e-849f-afbfd6911c69-cert\") pod \"kserve-controller-manager-674d959fb8-jpzhl\" (UID: \"13860f72-8141-4c7e-849f-afbfd6911c69\") " pod="kserve/kserve-controller-manager-674d959fb8-jpzhl"
Jul 31 23:52:12.008189 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:12.008150    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2ldmx" event={"ID":"ba2f18cd-6d28-4e57-aa42-f8d2f0e46f32","Type":"ContainerStarted","Data":"898b1f953ee766e039f4f2128419933b636920e73aed325905736c98f67dcda6"}
Jul 31 23:52:12.154841 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:12.154806    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-jpzhl"
Jul 31 23:52:12.299061 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:12.298954    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-jpzhl"]
Jul 31 23:52:12.419646 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:52:12.419608    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13860f72_8141_4c7e_849f_afbfd6911c69.slice/crio-a292ef63c00cb5d24c1a11b3f1c4e8eaf5806dbbbcd3afea593892cd7b2df4bf WatchSource:0}: Error finding container a292ef63c00cb5d24c1a11b3f1c4e8eaf5806dbbbcd3afea593892cd7b2df4bf: Status 404 returned error can't find the container with id a292ef63c00cb5d24c1a11b3f1c4e8eaf5806dbbbcd3afea593892cd7b2df4bf
Jul 31 23:52:13.014788 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:13.014752    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-jpzhl" event={"ID":"13860f72-8141-4c7e-849f-afbfd6911c69","Type":"ContainerStarted","Data":"a292ef63c00cb5d24c1a11b3f1c4e8eaf5806dbbbcd3afea593892cd7b2df4bf"}
Jul 31 23:52:17.035845 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:17.035802    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2ldmx" event={"ID":"ba2f18cd-6d28-4e57-aa42-f8d2f0e46f32","Type":"ContainerStarted","Data":"a52775f3bff720523f5a3799e1c9efe2e92f84f253eb1fc0bfb0494b6ae49e3a"}
Jul 31 23:52:17.036419 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:17.035909    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-2ldmx"
Jul 31 23:52:17.037467 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:17.037443    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-jpzhl" event={"ID":"13860f72-8141-4c7e-849f-afbfd6911c69","Type":"ContainerStarted","Data":"1ad48249cb6e64009189ca521c041c399306ccd144d29a6b806e00592a246573"}
Jul 31 23:52:17.037601 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:17.037587    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-674d959fb8-jpzhl"
Jul 31 23:52:17.086276 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:17.086228    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-2ldmx" podStartSLOduration=1.687971858 podStartE2EDuration="6.086212415s" totalImagesPullingTime="4.398240557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:17.061904664 +0000 UTC m=+371.935548273" watchObservedRunningTime="2026-07-31 23:52:17.086212415 +0000 UTC m=+371.959856025"
Jul 31 23:52:23.043672 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:23.043642    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-2ldmx"
Jul 31 23:52:23.061220 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:23.061164    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-674d959fb8-jpzhl" podStartSLOduration=8.444145998 podStartE2EDuration="12.061144021s" totalImagesPullingTime="3.616998023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:17.086018016 +0000 UTC m=+371.959661625" watchObservedRunningTime="2026-07-31 23:52:23.061144021 +0000 UTC m=+377.934787632"
Jul 31 23:52:48.046919 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:48.046889    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-674d959fb8-jpzhl"
Jul 31 23:52:54.845122 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:54.845092    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-jpzhl"]
Jul 31 23:52:54.845502 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:54.845301    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-674d959fb8-jpzhl" podUID="13860f72-8141-4c7e-849f-afbfd6911c69" containerName="manager" containerID="cri-o://1ad48249cb6e64009189ca521c041c399306ccd144d29a6b806e00592a246573" gracePeriod=10
Jul 31 23:52:54.866823 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:54.866780    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-md78t"]
Jul 31 23:52:54.868925 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:54.868906    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-md78t"
Jul 31 23:52:54.879711 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:54.879684    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-md78t"]
Jul 31 23:52:54.940234 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:54.940195    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn2rh\" (UniqueName: \"kubernetes.io/projected/35f2b516-698c-4978-95cc-c1500eafdbfe-kube-api-access-cn2rh\") pod \"kserve-controller-manager-674d959fb8-md78t\" (UID: \"35f2b516-698c-4978-95cc-c1500eafdbfe\") " pod="kserve/kserve-controller-manager-674d959fb8-md78t"
Jul 31 23:52:54.940380 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:54.940262    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35f2b516-698c-4978-95cc-c1500eafdbfe-cert\") pod \"kserve-controller-manager-674d959fb8-md78t\" (UID: \"35f2b516-698c-4978-95cc-c1500eafdbfe\") " pod="kserve/kserve-controller-manager-674d959fb8-md78t"
Jul 31 23:52:55.041444 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.041412    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn2rh\" (UniqueName: \"kubernetes.io/projected/35f2b516-698c-4978-95cc-c1500eafdbfe-kube-api-access-cn2rh\") pod \"kserve-controller-manager-674d959fb8-md78t\" (UID: \"35f2b516-698c-4978-95cc-c1500eafdbfe\") " pod="kserve/kserve-controller-manager-674d959fb8-md78t"
Jul 31 23:52:55.041602 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.041482    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35f2b516-698c-4978-95cc-c1500eafdbfe-cert\") pod \"kserve-controller-manager-674d959fb8-md78t\" (UID: \"35f2b516-698c-4978-95cc-c1500eafdbfe\") " pod="kserve/kserve-controller-manager-674d959fb8-md78t"
Jul 31 23:52:55.043649 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.043622    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35f2b516-698c-4978-95cc-c1500eafdbfe-cert\") pod \"kserve-controller-manager-674d959fb8-md78t\" (UID: \"35f2b516-698c-4978-95cc-c1500eafdbfe\") " pod="kserve/kserve-controller-manager-674d959fb8-md78t"
Jul 31 23:52:55.049326 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.049307    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn2rh\" (UniqueName: \"kubernetes.io/projected/35f2b516-698c-4978-95cc-c1500eafdbfe-kube-api-access-cn2rh\") pod \"kserve-controller-manager-674d959fb8-md78t\" (UID: \"35f2b516-698c-4978-95cc-c1500eafdbfe\") " pod="kserve/kserve-controller-manager-674d959fb8-md78t"
Jul 31 23:52:55.083355 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.083332    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-jpzhl"
Jul 31 23:52:55.141956 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.141876    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42q52\" (UniqueName: \"kubernetes.io/projected/13860f72-8141-4c7e-849f-afbfd6911c69-kube-api-access-42q52\") pod \"13860f72-8141-4c7e-849f-afbfd6911c69\" (UID: \"13860f72-8141-4c7e-849f-afbfd6911c69\") "
Jul 31 23:52:55.141956 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.141911    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13860f72-8141-4c7e-849f-afbfd6911c69-cert\") pod \"13860f72-8141-4c7e-849f-afbfd6911c69\" (UID: \"13860f72-8141-4c7e-849f-afbfd6911c69\") "
Jul 31 23:52:55.144042 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.144010    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13860f72-8141-4c7e-849f-afbfd6911c69-cert" (OuterVolumeSpecName: "cert") pod "13860f72-8141-4c7e-849f-afbfd6911c69" (UID: "13860f72-8141-4c7e-849f-afbfd6911c69"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:52:55.144042 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.144012    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13860f72-8141-4c7e-849f-afbfd6911c69-kube-api-access-42q52" (OuterVolumeSpecName: "kube-api-access-42q52") pod "13860f72-8141-4c7e-849f-afbfd6911c69" (UID: "13860f72-8141-4c7e-849f-afbfd6911c69"). InnerVolumeSpecName "kube-api-access-42q52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:52:55.191529 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.191502    2584 generic.go:356] "Generic (PLEG): container finished" podID="13860f72-8141-4c7e-849f-afbfd6911c69" containerID="1ad48249cb6e64009189ca521c041c399306ccd144d29a6b806e00592a246573" exitCode=0
Jul 31 23:52:55.191655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.191545    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-jpzhl"
Jul 31 23:52:55.191655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.191574    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-jpzhl" event={"ID":"13860f72-8141-4c7e-849f-afbfd6911c69","Type":"ContainerDied","Data":"1ad48249cb6e64009189ca521c041c399306ccd144d29a6b806e00592a246573"}
Jul 31 23:52:55.191655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.191617    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-jpzhl" event={"ID":"13860f72-8141-4c7e-849f-afbfd6911c69","Type":"ContainerDied","Data":"a292ef63c00cb5d24c1a11b3f1c4e8eaf5806dbbbcd3afea593892cd7b2df4bf"}
Jul 31 23:52:55.191655 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.191640    2584 scope.go:117] "RemoveContainer" containerID="1ad48249cb6e64009189ca521c041c399306ccd144d29a6b806e00592a246573"
Jul 31 23:52:55.200227 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.200193    2584 scope.go:117] "RemoveContainer" containerID="1ad48249cb6e64009189ca521c041c399306ccd144d29a6b806e00592a246573"
Jul 31 23:52:55.200473 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:52:55.200453    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad48249cb6e64009189ca521c041c399306ccd144d29a6b806e00592a246573\": container with ID starting with 1ad48249cb6e64009189ca521c041c399306ccd144d29a6b806e00592a246573 not found: ID does not exist" containerID="1ad48249cb6e64009189ca521c041c399306ccd144d29a6b806e00592a246573"
Jul 31 23:52:55.200550 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.200480    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad48249cb6e64009189ca521c041c399306ccd144d29a6b806e00592a246573"} err="failed to get container status \"1ad48249cb6e64009189ca521c041c399306ccd144d29a6b806e00592a246573\": rpc error: code = NotFound desc = could not find container \"1ad48249cb6e64009189ca521c041c399306ccd144d29a6b806e00592a246573\": container with ID starting with 1ad48249cb6e64009189ca521c041c399306ccd144d29a6b806e00592a246573 not found: ID does not exist"
Jul 31 23:52:55.212133 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.212112    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-jpzhl"]
Jul 31 23:52:55.215564 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.215545    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-jpzhl"]
Jul 31 23:52:55.242602 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.242585    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42q52\" (UniqueName: \"kubernetes.io/projected/13860f72-8141-4c7e-849f-afbfd6911c69-kube-api-access-42q52\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:52:55.242602 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.242603    2584 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13860f72-8141-4c7e-849f-afbfd6911c69-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:52:55.246563 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.246549    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-md78t"
Jul 31 23:52:55.368474 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.368451    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-md78t"]
Jul 31 23:52:55.370873 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:52:55.370849    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f2b516_698c_4978_95cc_c1500eafdbfe.slice/crio-8e85eed414905b99190d5de0f94d984db7d167493bb21f854a903dcd6f79bd76 WatchSource:0}: Error finding container 8e85eed414905b99190d5de0f94d984db7d167493bb21f854a903dcd6f79bd76: Status 404 returned error can't find the container with id 8e85eed414905b99190d5de0f94d984db7d167493bb21f854a903dcd6f79bd76
Jul 31 23:52:55.569889 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:55.569858    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13860f72-8141-4c7e-849f-afbfd6911c69" path="/var/lib/kubelet/pods/13860f72-8141-4c7e-849f-afbfd6911c69/volumes"
Jul 31 23:52:56.196799 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:56.196704    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-md78t" event={"ID":"35f2b516-698c-4978-95cc-c1500eafdbfe","Type":"ContainerStarted","Data":"3547580b539c67329cb69b3ad7327e3194fceef43b7fbd958d286525bab7f863"}
Jul 31 23:52:56.196799 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:56.196765    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-md78t" event={"ID":"35f2b516-698c-4978-95cc-c1500eafdbfe","Type":"ContainerStarted","Data":"8e85eed414905b99190d5de0f94d984db7d167493bb21f854a903dcd6f79bd76"}
Jul 31 23:52:56.196799 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:56.196798    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-674d959fb8-md78t"
Jul 31 23:52:56.212316 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:52:56.212275    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-674d959fb8-md78t" podStartSLOduration=1.849554408 podStartE2EDuration="2.212262298s" totalImagesPullingTime="362.70789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:56.211526857 +0000 UTC m=+411.085170466" watchObservedRunningTime="2026-07-31 23:52:56.212262298 +0000 UTC m=+411.085905908"
Jul 31 23:53:27.204864 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:27.204833    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-674d959fb8-md78t"
Jul 31 23:53:28.235182 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.235144    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-8588677cfb-482qx"]
Jul 31 23:53:28.235629 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.235573    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13860f72-8141-4c7e-849f-afbfd6911c69" containerName="manager"
Jul 31 23:53:28.235629 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.235592    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="13860f72-8141-4c7e-849f-afbfd6911c69" containerName="manager"
Jul 31 23:53:28.235746 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.235676    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="13860f72-8141-4c7e-849f-afbfd6911c69" containerName="manager"
Jul 31 23:53:28.238814 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.238794    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-482qx"
Jul 31 23:53:28.241264 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.241248    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 31 23:53:28.241390 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.241354    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-zcq4k\""
Jul 31 23:53:28.241390 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.241366    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Jul 31 23:53:28.247910 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.247888    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-482qx"]
Jul 31 23:53:28.292939 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.292913    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef58b45c-cd1b-4b63-b4bf-afee17ec88ff-cert\") pod \"odh-model-controller-8588677cfb-482qx\" (UID: \"ef58b45c-cd1b-4b63-b4bf-afee17ec88ff\") " pod="kserve/odh-model-controller-8588677cfb-482qx"
Jul 31 23:53:28.293079 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.292980    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef58b45c-cd1b-4b63-b4bf-afee17ec88ff-metrics-certs\") pod \"odh-model-controller-8588677cfb-482qx\" (UID: \"ef58b45c-cd1b-4b63-b4bf-afee17ec88ff\") " pod="kserve/odh-model-controller-8588677cfb-482qx"
Jul 31 23:53:28.293079 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.292999    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmqbs\" (UniqueName: \"kubernetes.io/projected/ef58b45c-cd1b-4b63-b4bf-afee17ec88ff-kube-api-access-nmqbs\") pod \"odh-model-controller-8588677cfb-482qx\" (UID: \"ef58b45c-cd1b-4b63-b4bf-afee17ec88ff\") " pod="kserve/odh-model-controller-8588677cfb-482qx"
Jul 31 23:53:28.393906 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.393875    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef58b45c-cd1b-4b63-b4bf-afee17ec88ff-cert\") pod \"odh-model-controller-8588677cfb-482qx\" (UID: \"ef58b45c-cd1b-4b63-b4bf-afee17ec88ff\") " pod="kserve/odh-model-controller-8588677cfb-482qx"
Jul 31 23:53:28.394088 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.393924    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef58b45c-cd1b-4b63-b4bf-afee17ec88ff-metrics-certs\") pod \"odh-model-controller-8588677cfb-482qx\" (UID: \"ef58b45c-cd1b-4b63-b4bf-afee17ec88ff\") " pod="kserve/odh-model-controller-8588677cfb-482qx"
Jul 31 23:53:28.394088 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.393941    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmqbs\" (UniqueName: \"kubernetes.io/projected/ef58b45c-cd1b-4b63-b4bf-afee17ec88ff-kube-api-access-nmqbs\") pod \"odh-model-controller-8588677cfb-482qx\" (UID: \"ef58b45c-cd1b-4b63-b4bf-afee17ec88ff\") " pod="kserve/odh-model-controller-8588677cfb-482qx"
Jul 31 23:53:28.396341 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.396320    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef58b45c-cd1b-4b63-b4bf-afee17ec88ff-metrics-certs\") pod \"odh-model-controller-8588677cfb-482qx\" (UID: \"ef58b45c-cd1b-4b63-b4bf-afee17ec88ff\") " pod="kserve/odh-model-controller-8588677cfb-482qx"
Jul 31 23:53:28.396341 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.396341    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef58b45c-cd1b-4b63-b4bf-afee17ec88ff-cert\") pod \"odh-model-controller-8588677cfb-482qx\" (UID: \"ef58b45c-cd1b-4b63-b4bf-afee17ec88ff\") " pod="kserve/odh-model-controller-8588677cfb-482qx"
Jul 31 23:53:28.401355 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.401336    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmqbs\" (UniqueName: \"kubernetes.io/projected/ef58b45c-cd1b-4b63-b4bf-afee17ec88ff-kube-api-access-nmqbs\") pod \"odh-model-controller-8588677cfb-482qx\" (UID: \"ef58b45c-cd1b-4b63-b4bf-afee17ec88ff\") " pod="kserve/odh-model-controller-8588677cfb-482qx"
Jul 31 23:53:28.548803 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.548767    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-482qx"
Jul 31 23:53:28.670033 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:28.669989    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-482qx"]
Jul 31 23:53:28.673611 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:53:28.673581    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef58b45c_cd1b_4b63_b4bf_afee17ec88ff.slice/crio-8bf412f83c329f46a860636f9af90e390c6f410da30ec25c954d35234c61f02d WatchSource:0}: Error finding container 8bf412f83c329f46a860636f9af90e390c6f410da30ec25c954d35234c61f02d: Status 404 returned error can't find the container with id 8bf412f83c329f46a860636f9af90e390c6f410da30ec25c954d35234c61f02d
Jul 31 23:53:29.328032 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:29.327997    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-482qx" event={"ID":"ef58b45c-cd1b-4b63-b4bf-afee17ec88ff","Type":"ContainerStarted","Data":"8bf412f83c329f46a860636f9af90e390c6f410da30ec25c954d35234c61f02d"}
Jul 31 23:53:32.341389 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:32.341354    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-482qx" event={"ID":"ef58b45c-cd1b-4b63-b4bf-afee17ec88ff","Type":"ContainerStarted","Data":"2c3037b9f66178d13d1435587725777ba9011943e3c9fea765d3b7da5a20559a"}
Jul 31 23:53:32.341817 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:32.341416    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-8588677cfb-482qx"
Jul 31 23:53:32.361190 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:32.361144    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-8588677cfb-482qx" podStartSLOduration=1.348656407 podStartE2EDuration="4.361130553s" totalImagesPullingTime="3.012474146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:53:32.360834507 +0000 UTC m=+447.234478117" watchObservedRunningTime="2026-07-31 23:53:32.361130553 +0000 UTC m=+447.234774163"
Jul 31 23:53:43.348401 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:53:43.348372    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-8588677cfb-482qx"
Jul 31 23:55:35.557327 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.557251    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-pjs54"]
Jul 31 23:55:35.560297 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.560281    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-pjs54"
Jul 31 23:55:35.562879 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.562852    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 31 23:55:35.562879 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.562879    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 31 23:55:35.572448 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.572426    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-pjs54"]
Jul 31 23:55:35.746294 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.746262    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57jfc\" (UniqueName: \"kubernetes.io/projected/1730bd87-0467-4e98-8651-8fdf1b6a995b-kube-api-access-57jfc\") pod \"seaweedfs-tls-custom-8b44d767d-pjs54\" (UID: \"1730bd87-0467-4e98-8651-8fdf1b6a995b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-pjs54"
Jul 31 23:55:35.746438 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.746327    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/1730bd87-0467-4e98-8651-8fdf1b6a995b-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-pjs54\" (UID: \"1730bd87-0467-4e98-8651-8fdf1b6a995b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-pjs54"
Jul 31 23:55:35.746438 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.746355    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1730bd87-0467-4e98-8651-8fdf1b6a995b-data\") pod \"seaweedfs-tls-custom-8b44d767d-pjs54\" (UID: \"1730bd87-0467-4e98-8651-8fdf1b6a995b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-pjs54"
Jul 31 23:55:35.847139 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.847063    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/1730bd87-0467-4e98-8651-8fdf1b6a995b-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-pjs54\" (UID: \"1730bd87-0467-4e98-8651-8fdf1b6a995b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-pjs54"
Jul 31 23:55:35.847139 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.847098    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1730bd87-0467-4e98-8651-8fdf1b6a995b-data\") pod \"seaweedfs-tls-custom-8b44d767d-pjs54\" (UID: \"1730bd87-0467-4e98-8651-8fdf1b6a995b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-pjs54"
Jul 31 23:55:35.847139 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.847136    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jfc\" (UniqueName: \"kubernetes.io/projected/1730bd87-0467-4e98-8651-8fdf1b6a995b-kube-api-access-57jfc\") pod \"seaweedfs-tls-custom-8b44d767d-pjs54\" (UID: \"1730bd87-0467-4e98-8651-8fdf1b6a995b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-pjs54"
Jul 31 23:55:35.847584 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.847558    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1730bd87-0467-4e98-8651-8fdf1b6a995b-data\") pod \"seaweedfs-tls-custom-8b44d767d-pjs54\" (UID: \"1730bd87-0467-4e98-8651-8fdf1b6a995b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-pjs54"
Jul 31 23:55:35.849477 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.849452    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/1730bd87-0467-4e98-8651-8fdf1b6a995b-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-pjs54\" (UID: \"1730bd87-0467-4e98-8651-8fdf1b6a995b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-pjs54"
Jul 31 23:55:35.855345 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.855324    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57jfc\" (UniqueName: \"kubernetes.io/projected/1730bd87-0467-4e98-8651-8fdf1b6a995b-kube-api-access-57jfc\") pod \"seaweedfs-tls-custom-8b44d767d-pjs54\" (UID: \"1730bd87-0467-4e98-8651-8fdf1b6a995b\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-pjs54"
Jul 31 23:55:35.870347 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.870329    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-pjs54"
Jul 31 23:55:35.992071 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:35.992004    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-pjs54"]
Jul 31 23:55:35.994184 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:55:35.994152    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1730bd87_0467_4e98_8651_8fdf1b6a995b.slice/crio-e111317228cceb6cef3ecaab31699eca52244b21e46a5f5169301ac0f9388c12 WatchSource:0}: Error finding container e111317228cceb6cef3ecaab31699eca52244b21e46a5f5169301ac0f9388c12: Status 404 returned error can't find the container with id e111317228cceb6cef3ecaab31699eca52244b21e46a5f5169301ac0f9388c12
Jul 31 23:55:36.843914 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:36.843876    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-pjs54" event={"ID":"1730bd87-0467-4e98-8651-8fdf1b6a995b","Type":"ContainerStarted","Data":"d5d810e0cb034a8b4e7f15809cba7664e8deac1f09c917fa2021194001d87aa1"}
Jul 31 23:55:36.844324 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:36.843931    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-pjs54" event={"ID":"1730bd87-0467-4e98-8651-8fdf1b6a995b","Type":"ContainerStarted","Data":"e111317228cceb6cef3ecaab31699eca52244b21e46a5f5169301ac0f9388c12"}
Jul 31 23:55:36.860590 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:36.860544    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-pjs54" podStartSLOduration=1.605470763 podStartE2EDuration="1.860531087s" totalImagesPullingTime="255.060324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:55:36.859262187 +0000 UTC m=+571.732905798" watchObservedRunningTime="2026-07-31 23:55:36.860531087 +0000 UTC m=+571.734174697"
Jul 31 23:55:48.594914 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:48.594883    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9"]
Jul 31 23:55:48.599014 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:48.598997    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9"
Jul 31 23:55:48.601668 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:48.601650    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 31 23:55:48.601763 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:48.601687    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 31 23:55:48.606102 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:48.606062    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9"]
Jul 31 23:55:48.642441 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:48.642410    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz4hf\" (UniqueName: \"kubernetes.io/projected/90812f05-db47-4035-93e8-d821f3723f3f-kube-api-access-pz4hf\") pod \"seaweedfs-tls-serving-67b7477c6d-x6lk9\" (UID: \"90812f05-db47-4035-93e8-d821f3723f3f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9"
Jul 31 23:55:48.642586 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:48.642457    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/90812f05-db47-4035-93e8-d821f3723f3f-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-x6lk9\" (UID: \"90812f05-db47-4035-93e8-d821f3723f3f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9"
Jul 31 23:55:48.642586 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:48.642488    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/90812f05-db47-4035-93e8-d821f3723f3f-data\") pod \"seaweedfs-tls-serving-67b7477c6d-x6lk9\" (UID: \"90812f05-db47-4035-93e8-d821f3723f3f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9"
Jul 31 23:55:48.743802 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:48.743765    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz4hf\" (UniqueName: \"kubernetes.io/projected/90812f05-db47-4035-93e8-d821f3723f3f-kube-api-access-pz4hf\") pod \"seaweedfs-tls-serving-67b7477c6d-x6lk9\" (UID: \"90812f05-db47-4035-93e8-d821f3723f3f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9"
Jul 31 23:55:48.743996 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:48.743826    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/90812f05-db47-4035-93e8-d821f3723f3f-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-x6lk9\" (UID: \"90812f05-db47-4035-93e8-d821f3723f3f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9"
Jul 31 23:55:48.743996 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:48.743859    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/90812f05-db47-4035-93e8-d821f3723f3f-data\") pod \"seaweedfs-tls-serving-67b7477c6d-x6lk9\" (UID: \"90812f05-db47-4035-93e8-d821f3723f3f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9"
Jul 31 23:55:48.744213 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:55:48.744001    2584 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jul 31 23:55:48.744213 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:55:48.744025    2584 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9: secret "seaweedfs-tls-serving" not found
Jul 31 23:55:48.744213 ip-10-0-137-252 kubenswrapper[2584]: E0731 23:55:48.744093    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90812f05-db47-4035-93e8-d821f3723f3f-seaweedfs-tls-serving podName:90812f05-db47-4035-93e8-d821f3723f3f nodeName:}" failed. No retries permitted until 2026-07-31 23:55:49.244075491 +0000 UTC m=+584.117719090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/90812f05-db47-4035-93e8-d821f3723f3f-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-x6lk9" (UID: "90812f05-db47-4035-93e8-d821f3723f3f") : secret "seaweedfs-tls-serving" not found
Jul 31 23:55:48.744389 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:48.744365    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/90812f05-db47-4035-93e8-d821f3723f3f-data\") pod \"seaweedfs-tls-serving-67b7477c6d-x6lk9\" (UID: \"90812f05-db47-4035-93e8-d821f3723f3f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9"
Jul 31 23:55:48.752531 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:48.752507    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz4hf\" (UniqueName: \"kubernetes.io/projected/90812f05-db47-4035-93e8-d821f3723f3f-kube-api-access-pz4hf\") pod \"seaweedfs-tls-serving-67b7477c6d-x6lk9\" (UID: \"90812f05-db47-4035-93e8-d821f3723f3f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9"
Jul 31 23:55:49.248423 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:49.248394    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/90812f05-db47-4035-93e8-d821f3723f3f-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-x6lk9\" (UID: \"90812f05-db47-4035-93e8-d821f3723f3f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9"
Jul 31 23:55:49.250729 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:49.250697    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/90812f05-db47-4035-93e8-d821f3723f3f-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-x6lk9\" (UID: \"90812f05-db47-4035-93e8-d821f3723f3f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9"
Jul 31 23:55:49.508632 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:49.508548    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9"
Jul 31 23:55:49.638290 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:49.638126    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9"]
Jul 31 23:55:49.641231 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:55:49.641201    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90812f05_db47_4035_93e8_d821f3723f3f.slice/crio-20ca396d9b61737f212c74f8157cef103a4a7d1d4ac0ba3bbd3abbe046ad820b WatchSource:0}: Error finding container 20ca396d9b61737f212c74f8157cef103a4a7d1d4ac0ba3bbd3abbe046ad820b: Status 404 returned error can't find the container with id 20ca396d9b61737f212c74f8157cef103a4a7d1d4ac0ba3bbd3abbe046ad820b
Jul 31 23:55:49.896625 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:49.896596    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9" event={"ID":"90812f05-db47-4035-93e8-d821f3723f3f","Type":"ContainerStarted","Data":"20ca396d9b61737f212c74f8157cef103a4a7d1d4ac0ba3bbd3abbe046ad820b"}
Jul 31 23:55:50.901674 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:50.901635    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9" event={"ID":"90812f05-db47-4035-93e8-d821f3723f3f","Type":"ContainerStarted","Data":"f9552cce34ca98d389311806ca60cccb5f7b2b9be3e67d8f809b418f72aba326"}
Jul 31 23:55:50.918062 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:55:50.918014    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-x6lk9" podStartSLOduration=2.66491882 podStartE2EDuration="2.917998439s" totalImagesPullingTime="253.079619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:55:50.917393014 +0000 UTC m=+585.791036637" watchObservedRunningTime="2026-07-31 23:55:50.917998439 +0000 UTC m=+585.791642049"
Jul 31 23:56:05.518369 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:05.518332    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Jul 31 23:56:05.519762 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:05.519738    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Jul 31 23:56:15.044234 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.044187    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"]
Jul 31 23:56:15.046626 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.046607    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.049621 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.049597    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 23:56:15.049742 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.049620    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 23:56:15.049742 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.049647    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 23:56:15.049742 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.049597    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 31 23:56:15.049742 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.049600    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Jul 31 23:56:15.050455 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.050436    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 31 23:56:15.050623 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.050603    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 23:56:15.058495 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.058468    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"]
Jul 31 23:56:15.157251 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.157211    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.157444 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.157260    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.157444 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.157356    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.157561 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.157448    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cktgv\" (UniqueName: \"kubernetes.io/projected/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kube-api-access-cktgv\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.157561 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.157484    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.258639 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.258605    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.258639 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.258642    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.258849 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.258676    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.258849 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.258720    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktgv\" (UniqueName: \"kubernetes.io/projected/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kube-api-access-cktgv\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.258849 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.258741    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.259242 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.259219    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.259444 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.259426    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.259573 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.259553    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.261240 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.261222    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.266923 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.266901    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cktgv\" (UniqueName: \"kubernetes.io/projected/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kube-api-access-cktgv\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.359287 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.359188    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:15.490596 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:15.490570    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"]
Jul 31 23:56:15.492825 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:56:15.492796    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131577d3_e2cf_4b17_aa60_8f8f6dd6b6fd.slice/crio-d959f5d29c6936cfa4b46f3a6638f53f4231c68ccfb0818ae083ff3ad82ebbaf WatchSource:0}: Error finding container d959f5d29c6936cfa4b46f3a6638f53f4231c68ccfb0818ae083ff3ad82ebbaf: Status 404 returned error can't find the container with id d959f5d29c6936cfa4b46f3a6638f53f4231c68ccfb0818ae083ff3ad82ebbaf
Jul 31 23:56:16.010482 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:16.010442    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" event={"ID":"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd","Type":"ContainerStarted","Data":"d959f5d29c6936cfa4b46f3a6638f53f4231c68ccfb0818ae083ff3ad82ebbaf"}
Jul 31 23:56:19.025232 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:19.025195    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" event={"ID":"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd","Type":"ContainerStarted","Data":"4f0a1e7c92233e8fa68097c0a34b40783947c758c182d0d1bee15d5b33308c98"}
Jul 31 23:56:22.038800 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:22.038763    2584 generic.go:356] "Generic (PLEG): container finished" podID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerID="4f0a1e7c92233e8fa68097c0a34b40783947c758c182d0d1bee15d5b33308c98" exitCode=0
Jul 31 23:56:22.039205 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:22.038833    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" event={"ID":"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd","Type":"ContainerDied","Data":"4f0a1e7c92233e8fa68097c0a34b40783947c758c182d0d1bee15d5b33308c98"}
Jul 31 23:56:29.077632 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:29.077576    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" event={"ID":"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd","Type":"ContainerStarted","Data":"9aa7a00ef653c430699bc8573da0039914a7e6408e286775fbecc278d7eb32f9"}
Jul 31 23:56:32.093032 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:32.092991    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" event={"ID":"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd","Type":"ContainerStarted","Data":"e8502ce3159dd12bd84614bd680bafed36f4e7f2720581364cc975eeeaf84b98"}
Jul 31 23:56:32.093412 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:32.093130    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:32.112249 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:32.112205    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" podStartSLOduration=1.6665573280000001 podStartE2EDuration="17.112183604s" totalImagesPullingTime="11.986538664s" totalInitContainerRuntime="3.459087612s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:56:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:56:32.109833494 +0000 UTC m=+626.983477103" watchObservedRunningTime="2026-07-31 23:56:32.112183604 +0000 UTC m=+626.985827214"
Jul 31 23:56:33.097846 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:33.097810    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:33.099204 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:33.099170    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 31 23:56:34.101589 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:34.101557    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 31 23:56:39.106207 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:39.106174    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:56:39.106609 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:39.106584    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 31 23:56:49.107453 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:49.107412    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 31 23:56:59.106852 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:56:59.106808    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 31 23:57:09.107025 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:09.106986    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 31 23:57:19.107517 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:19.107478    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 31 23:57:29.107382 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:29.107342    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 31 23:57:39.106929 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:39.106888    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 31 23:57:49.107119 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:49.107084    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:57:56.086758 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:56.086723    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"]
Jul 31 23:57:56.087312 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:56.086954    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kserve-container" containerID="cri-o://9aa7a00ef653c430699bc8573da0039914a7e6408e286775fbecc278d7eb32f9" gracePeriod=30
Jul 31 23:57:56.087312 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:56.087007    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kube-rbac-proxy" containerID="cri-o://e8502ce3159dd12bd84614bd680bafed36f4e7f2720581364cc975eeeaf84b98" gracePeriod=30
Jul 31 23:57:56.456656 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:56.456554    2584 generic.go:356] "Generic (PLEG): container finished" podID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerID="e8502ce3159dd12bd84614bd680bafed36f4e7f2720581364cc975eeeaf84b98" exitCode=2
Jul 31 23:57:56.456656 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:56.456602    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" event={"ID":"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd","Type":"ContainerDied","Data":"e8502ce3159dd12bd84614bd680bafed36f4e7f2720581364cc975eeeaf84b98"}
Jul 31 23:57:59.102348 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.102304    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 31 23:57:59.106632 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.106607    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 31 23:57:59.472481 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.472449    2584 generic.go:356] "Generic (PLEG): container finished" podID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerID="9aa7a00ef653c430699bc8573da0039914a7e6408e286775fbecc278d7eb32f9" exitCode=0
Jul 31 23:57:59.472634 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.472510    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" event={"ID":"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd","Type":"ContainerDied","Data":"9aa7a00ef653c430699bc8573da0039914a7e6408e286775fbecc278d7eb32f9"}
Jul 31 23:57:59.523826 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.523800    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:57:59.589449 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.589409    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kube-rbac-proxy-sar-config\") pod \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") "
Jul 31 23:57:59.589449 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.589452    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-proxy-tls\") pod \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") "
Jul 31 23:57:59.589672 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.589482    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cktgv\" (UniqueName: \"kubernetes.io/projected/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kube-api-access-cktgv\") pod \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") "
Jul 31 23:57:59.589672 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.589512    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-cabundle-cert\") pod \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") "
Jul 31 23:57:59.589672 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.589534    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kserve-provision-location\") pod \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\" (UID: \"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd\") "
Jul 31 23:57:59.589833 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.589781    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" (UID: "131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:57:59.589915 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.589880    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" (UID: "131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:57:59.589957 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.589923    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" (UID: "131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:57:59.591643 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.591625    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" (UID: "131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:57:59.591737 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.591711    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kube-api-access-cktgv" (OuterVolumeSpecName: "kube-api-access-cktgv") pod "131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" (UID: "131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd"). InnerVolumeSpecName "kube-api-access-cktgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:57:59.690588 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.690509    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:57:59.690588 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.690540    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:57:59.690588 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.690555    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:57:59.690588 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.690564    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cktgv\" (UniqueName: \"kubernetes.io/projected/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-kube-api-access-cktgv\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:57:59.690588 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:57:59.690573    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Jul 31 23:58:00.478979 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:58:00.478946    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"
Jul 31 23:58:00.479395 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:58:00.478945    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv" event={"ID":"131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd","Type":"ContainerDied","Data":"d959f5d29c6936cfa4b46f3a6638f53f4231c68ccfb0818ae083ff3ad82ebbaf"}
Jul 31 23:58:00.479395 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:58:00.479107    2584 scope.go:117] "RemoveContainer" containerID="e8502ce3159dd12bd84614bd680bafed36f4e7f2720581364cc975eeeaf84b98"
Jul 31 23:58:00.489736 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:58:00.489710    2584 scope.go:117] "RemoveContainer" containerID="9aa7a00ef653c430699bc8573da0039914a7e6408e286775fbecc278d7eb32f9"
Jul 31 23:58:00.498421 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:58:00.498401    2584 scope.go:117] "RemoveContainer" containerID="4f0a1e7c92233e8fa68097c0a34b40783947c758c182d0d1bee15d5b33308c98"
Jul 31 23:58:00.504936 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:58:00.504913    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"]
Jul 31 23:58:00.510129 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:58:00.510107    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jwbpv"]
Jul 31 23:58:01.570958 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:58:01.570927    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" path="/var/lib/kubelet/pods/131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd/volumes"
Jul 31 23:59:35.269980 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.269877    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"]
Jul 31 23:59:35.270534 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.270239    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="storage-initializer"
Jul 31 23:59:35.270534 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.270254    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="storage-initializer"
Jul 31 23:59:35.270534 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.270267    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kube-rbac-proxy"
Jul 31 23:59:35.270534 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.270273    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kube-rbac-proxy"
Jul 31 23:59:35.270534 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.270292    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kserve-container"
Jul 31 23:59:35.270534 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.270301    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kserve-container"
Jul 31 23:59:35.270534 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.270371    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kserve-container"
Jul 31 23:59:35.270534 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.270381    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="131577d3-e2cf-4b17-aa60-8f8f6dd6b6fd" containerName="kube-rbac-proxy"
Jul 31 23:59:35.272339 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.272318    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Jul 31 23:59:35.275096 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.275071    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Jul 31 23:59:35.275228 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.275137    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 31 23:59:35.275228 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.275146    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 31 23:59:35.276222 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.276186    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 23:59:35.276341 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.276188    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 23:59:35.283240 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.283217    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"]
Jul 31 23:59:35.355250 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.355216    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-n7rff\" (UID: \"b9d7aa3d-9b73-459a-a83b-8f3963caa28b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Jul 31 23:59:35.355444 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.355267    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-n7rff\" (UID: \"b9d7aa3d-9b73-459a-a83b-8f3963caa28b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Jul 31 23:59:35.355444 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.355284    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwgm9\" (UniqueName: \"kubernetes.io/projected/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-kube-api-access-kwgm9\") pod \"message-dumper-predictor-6575bcfdfd-n7rff\" (UID: \"b9d7aa3d-9b73-459a-a83b-8f3963caa28b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Jul 31 23:59:35.456259 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.456226    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-n7rff\" (UID: \"b9d7aa3d-9b73-459a-a83b-8f3963caa28b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Jul 31 23:59:35.456420 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.456272    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-n7rff\" (UID: \"b9d7aa3d-9b73-459a-a83b-8f3963caa28b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Jul 31 23:59:35.456420 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.456289    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwgm9\" (UniqueName: \"kubernetes.io/projected/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-kube-api-access-kwgm9\") pod \"message-dumper-predictor-6575bcfdfd-n7rff\" (UID: \"b9d7aa3d-9b73-459a-a83b-8f3963caa28b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Jul 31 23:59:35.457047 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.457016    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-n7rff\" (UID: \"b9d7aa3d-9b73-459a-a83b-8f3963caa28b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Jul 31 23:59:35.458694 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.458672    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-n7rff\" (UID: \"b9d7aa3d-9b73-459a-a83b-8f3963caa28b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Jul 31 23:59:35.463779 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.463751    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwgm9\" (UniqueName: \"kubernetes.io/projected/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-kube-api-access-kwgm9\") pod \"message-dumper-predictor-6575bcfdfd-n7rff\" (UID: \"b9d7aa3d-9b73-459a-a83b-8f3963caa28b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Jul 31 23:59:35.584930 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.584899    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Jul 31 23:59:35.708452 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.708428    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"]
Jul 31 23:59:35.710225 ip-10-0-137-252 kubenswrapper[2584]: W0731 23:59:35.710198    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d7aa3d_9b73_459a_a83b_8f3963caa28b.slice/crio-62008c43334ce94ed8cc7e043b6061557e54b93bae1c01ce1e2539882dd2a49f WatchSource:0}: Error finding container 62008c43334ce94ed8cc7e043b6061557e54b93bae1c01ce1e2539882dd2a49f: Status 404 returned error can't find the container with id 62008c43334ce94ed8cc7e043b6061557e54b93bae1c01ce1e2539882dd2a49f
Jul 31 23:59:35.711975 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.711950    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:59:35.853825 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:35.853753    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff" event={"ID":"b9d7aa3d-9b73-459a-a83b-8f3963caa28b","Type":"ContainerStarted","Data":"62008c43334ce94ed8cc7e043b6061557e54b93bae1c01ce1e2539882dd2a49f"}
Jul 31 23:59:37.864590 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:37.864554    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff" event={"ID":"b9d7aa3d-9b73-459a-a83b-8f3963caa28b","Type":"ContainerStarted","Data":"81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc"}
Jul 31 23:59:37.865089 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:37.864624    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Jul 31 23:59:37.865089 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:37.864652    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff" event={"ID":"b9d7aa3d-9b73-459a-a83b-8f3963caa28b","Type":"ContainerStarted","Data":"329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79"}
Jul 31 23:59:37.865089 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:37.864684    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Jul 31 23:59:37.866331 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:37.866311    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Jul 31 23:59:37.883696 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:37.883652    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff" podStartSLOduration=1.468261896 podStartE2EDuration="2.883636309s" totalImagesPullingTime="1.415374413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:59:37.88340355 +0000 UTC m=+812.757047160" watchObservedRunningTime="2026-07-31 23:59:37.883636309 +0000 UTC m=+812.757279922"
Jul 31 23:59:44.878552 ip-10-0-137-252 kubenswrapper[2584]: I0731 23:59:44.878519    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Aug 01 00:00:00.136342 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.136306    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29759040-dmdz6"]
Aug 01 00:00:00.143250 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.143229    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29759040-dmdz6"
Aug 01 00:00:00.146148 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.146124    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-zkdgq\""
Aug 01 00:00:00.146267 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.146125    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\""
Aug 01 00:00:00.152868 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.152848    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29759040-dmdz6"]
Aug 01 00:00:00.158911 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.158844    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/98479c90-4fef-4b82-af4f-d9bb7ab6fa6c-serviceca\") pod \"image-pruner-29759040-dmdz6\" (UID: \"98479c90-4fef-4b82-af4f-d9bb7ab6fa6c\") " pod="openshift-image-registry/image-pruner-29759040-dmdz6"
Aug 01 00:00:00.159001 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.158909    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86lmk\" (UniqueName: \"kubernetes.io/projected/98479c90-4fef-4b82-af4f-d9bb7ab6fa6c-kube-api-access-86lmk\") pod \"image-pruner-29759040-dmdz6\" (UID: \"98479c90-4fef-4b82-af4f-d9bb7ab6fa6c\") " pod="openshift-image-registry/image-pruner-29759040-dmdz6"
Aug 01 00:00:00.260191 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.260158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/98479c90-4fef-4b82-af4f-d9bb7ab6fa6c-serviceca\") pod \"image-pruner-29759040-dmdz6\" (UID: \"98479c90-4fef-4b82-af4f-d9bb7ab6fa6c\") " pod="openshift-image-registry/image-pruner-29759040-dmdz6"
Aug 01 00:00:00.260355 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.260209    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86lmk\" (UniqueName: \"kubernetes.io/projected/98479c90-4fef-4b82-af4f-d9bb7ab6fa6c-kube-api-access-86lmk\") pod \"image-pruner-29759040-dmdz6\" (UID: \"98479c90-4fef-4b82-af4f-d9bb7ab6fa6c\") " pod="openshift-image-registry/image-pruner-29759040-dmdz6"
Aug 01 00:00:00.260744 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.260725    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/98479c90-4fef-4b82-af4f-d9bb7ab6fa6c-serviceca\") pod \"image-pruner-29759040-dmdz6\" (UID: \"98479c90-4fef-4b82-af4f-d9bb7ab6fa6c\") " pod="openshift-image-registry/image-pruner-29759040-dmdz6"
Aug 01 00:00:00.269690 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.269663    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86lmk\" (UniqueName: \"kubernetes.io/projected/98479c90-4fef-4b82-af4f-d9bb7ab6fa6c-kube-api-access-86lmk\") pod \"image-pruner-29759040-dmdz6\" (UID: \"98479c90-4fef-4b82-af4f-d9bb7ab6fa6c\") " pod="openshift-image-registry/image-pruner-29759040-dmdz6"
Aug 01 00:00:00.470166 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.470101    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29759040-dmdz6"
Aug 01 00:00:00.595595 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.595573    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29759040-dmdz6"]
Aug 01 00:00:00.598137 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:00:00.598109    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98479c90_4fef_4b82_af4f_d9bb7ab6fa6c.slice/crio-54b9e7746124162e1a2f06489107517d9a89f79485bc76d14d0dcb57321f4adf WatchSource:0}: Error finding container 54b9e7746124162e1a2f06489107517d9a89f79485bc76d14d0dcb57321f4adf: Status 404 returned error can't find the container with id 54b9e7746124162e1a2f06489107517d9a89f79485bc76d14d0dcb57321f4adf
Aug 01 00:00:00.970508 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.970475    2584 generic.go:356] "Generic (PLEG): container finished" podID="98479c90-4fef-4b82-af4f-d9bb7ab6fa6c" containerID="0a043edc0a10c8874e0b00e51cc9c6e17f6e374117fc031461ecc0e1adec4c3f" exitCode=0
Aug 01 00:00:00.970616 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.970526    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29759040-dmdz6" event={"ID":"98479c90-4fef-4b82-af4f-d9bb7ab6fa6c","Type":"ContainerDied","Data":"0a043edc0a10c8874e0b00e51cc9c6e17f6e374117fc031461ecc0e1adec4c3f"}
Aug 01 00:00:00.970616 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:00.970570    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29759040-dmdz6" event={"ID":"98479c90-4fef-4b82-af4f-d9bb7ab6fa6c","Type":"ContainerStarted","Data":"54b9e7746124162e1a2f06489107517d9a89f79485bc76d14d0dcb57321f4adf"}
Aug 01 00:00:02.100427 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:02.100405    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29759040-dmdz6"
Aug 01 00:00:02.174698 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:02.174665    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86lmk\" (UniqueName: \"kubernetes.io/projected/98479c90-4fef-4b82-af4f-d9bb7ab6fa6c-kube-api-access-86lmk\") pod \"98479c90-4fef-4b82-af4f-d9bb7ab6fa6c\" (UID: \"98479c90-4fef-4b82-af4f-d9bb7ab6fa6c\") "
Aug 01 00:00:02.174863 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:02.174738    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/98479c90-4fef-4b82-af4f-d9bb7ab6fa6c-serviceca\") pod \"98479c90-4fef-4b82-af4f-d9bb7ab6fa6c\" (UID: \"98479c90-4fef-4b82-af4f-d9bb7ab6fa6c\") "
Aug 01 00:00:02.175156 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:02.175128    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98479c90-4fef-4b82-af4f-d9bb7ab6fa6c-serviceca" (OuterVolumeSpecName: "serviceca") pod "98479c90-4fef-4b82-af4f-d9bb7ab6fa6c" (UID: "98479c90-4fef-4b82-af4f-d9bb7ab6fa6c"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:02.176728 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:02.176703    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98479c90-4fef-4b82-af4f-d9bb7ab6fa6c-kube-api-access-86lmk" (OuterVolumeSpecName: "kube-api-access-86lmk") pod "98479c90-4fef-4b82-af4f-d9bb7ab6fa6c" (UID: "98479c90-4fef-4b82-af4f-d9bb7ab6fa6c"). InnerVolumeSpecName "kube-api-access-86lmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:00:02.276042 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:02.275924    2584 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/98479c90-4fef-4b82-af4f-d9bb7ab6fa6c-serviceca\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:00:02.276042 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:02.275956    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86lmk\" (UniqueName: \"kubernetes.io/projected/98479c90-4fef-4b82-af4f-d9bb7ab6fa6c-kube-api-access-86lmk\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:00:02.979887 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:02.979848    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29759040-dmdz6" event={"ID":"98479c90-4fef-4b82-af4f-d9bb7ab6fa6c","Type":"ContainerDied","Data":"54b9e7746124162e1a2f06489107517d9a89f79485bc76d14d0dcb57321f4adf"}
Aug 01 00:00:02.979887 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:02.979869    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29759040-dmdz6"
Aug 01 00:00:02.979887 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:02.979896    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54b9e7746124162e1a2f06489107517d9a89f79485bc76d14d0dcb57321f4adf"
Aug 01 00:00:22.940052 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:22.940020    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"]
Aug 01 00:00:22.940537 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:22.940337    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98479c90-4fef-4b82-af4f-d9bb7ab6fa6c" containerName="image-pruner"
Aug 01 00:00:22.940537 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:22.940348    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="98479c90-4fef-4b82-af4f-d9bb7ab6fa6c" containerName="image-pruner"
Aug 01 00:00:22.940537 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:22.940399    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="98479c90-4fef-4b82-af4f-d9bb7ab6fa6c" containerName="image-pruner"
Aug 01 00:00:22.943901 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:22.943879    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:22.946341 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:22.946323    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Aug 01 00:00:22.946455 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:22.946323    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:00:22.946667 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:22.946651    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:00:22.946744 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:22.946656    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Aug 01 00:00:22.954765 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:22.954741    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"]
Aug 01 00:00:23.041326 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.041298    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44dadffc-1262-4294-b122-19e4da11a49d-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-tc7qv\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.041466 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.041336    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44dadffc-1262-4294-b122-19e4da11a49d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-tc7qv\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.041466 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.041373    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44dadffc-1262-4294-b122-19e4da11a49d-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-tc7qv\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.041466 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.041432    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44dadffc-1262-4294-b122-19e4da11a49d-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-tc7qv\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.041572 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.041477    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmxkb\" (UniqueName: \"kubernetes.io/projected/44dadffc-1262-4294-b122-19e4da11a49d-kube-api-access-xmxkb\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-tc7qv\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.142792 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.142761    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44dadffc-1262-4294-b122-19e4da11a49d-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-tc7qv\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.142792 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.142794    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44dadffc-1262-4294-b122-19e4da11a49d-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-tc7qv\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.143024 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.142822    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmxkb\" (UniqueName: \"kubernetes.io/projected/44dadffc-1262-4294-b122-19e4da11a49d-kube-api-access-xmxkb\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-tc7qv\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.143024 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.142845    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44dadffc-1262-4294-b122-19e4da11a49d-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-tc7qv\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.143024 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.142877    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44dadffc-1262-4294-b122-19e4da11a49d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-tc7qv\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.143283 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.143261    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44dadffc-1262-4294-b122-19e4da11a49d-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-tc7qv\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.143638 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.143621    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44dadffc-1262-4294-b122-19e4da11a49d-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-tc7qv\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.143674 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.143623    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44dadffc-1262-4294-b122-19e4da11a49d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-tc7qv\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.145308 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.145286    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44dadffc-1262-4294-b122-19e4da11a49d-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-tc7qv\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.156767 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.156746    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmxkb\" (UniqueName: \"kubernetes.io/projected/44dadffc-1262-4294-b122-19e4da11a49d-kube-api-access-xmxkb\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-tc7qv\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.255579 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.255504    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:23.382949 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:23.382925    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"]
Aug 01 00:00:23.384803 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:00:23.384776    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44dadffc_1262_4294_b122_19e4da11a49d.slice/crio-9e32c4793027d4b7e80ddbb5942649366abf1a5be755628aa2632ed9eaee69c4 WatchSource:0}: Error finding container 9e32c4793027d4b7e80ddbb5942649366abf1a5be755628aa2632ed9eaee69c4: Status 404 returned error can't find the container with id 9e32c4793027d4b7e80ddbb5942649366abf1a5be755628aa2632ed9eaee69c4
Aug 01 00:00:24.078031 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:24.077989    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" event={"ID":"44dadffc-1262-4294-b122-19e4da11a49d","Type":"ContainerStarted","Data":"2bd54f1d5a2e3ae775de838100f1e733ea974a3d28ab48c04b207b38b48ea7d3"}
Aug 01 00:00:24.078425 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:24.078096    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" event={"ID":"44dadffc-1262-4294-b122-19e4da11a49d","Type":"ContainerStarted","Data":"9e32c4793027d4b7e80ddbb5942649366abf1a5be755628aa2632ed9eaee69c4"}
Aug 01 00:00:28.095827 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:28.095791    2584 generic.go:356] "Generic (PLEG): container finished" podID="44dadffc-1262-4294-b122-19e4da11a49d" containerID="2bd54f1d5a2e3ae775de838100f1e733ea974a3d28ab48c04b207b38b48ea7d3" exitCode=0
Aug 01 00:00:28.096342 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:28.095856    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" event={"ID":"44dadffc-1262-4294-b122-19e4da11a49d","Type":"ContainerDied","Data":"2bd54f1d5a2e3ae775de838100f1e733ea974a3d28ab48c04b207b38b48ea7d3"}
Aug 01 00:00:50.220760 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:50.220720    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" event={"ID":"44dadffc-1262-4294-b122-19e4da11a49d","Type":"ContainerStarted","Data":"5f43bd2047fa5620b8cdf56550ceb9f4178a6832a3efabe43195ff7df486b416"}
Aug 01 00:00:50.221175 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:50.220840    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" event={"ID":"44dadffc-1262-4294-b122-19e4da11a49d","Type":"ContainerStarted","Data":"959e3bea7e5ce537c2719ea50f6ed36091ba5c6fd9e8bb89aeed234d984ca523"}
Aug 01 00:00:50.221175 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:50.221154    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:50.221286 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:50.221209    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:50.222193 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:50.222161    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 01 00:00:50.241687 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:50.241641    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" podStartSLOduration=3.37757216 podStartE2EDuration="28.241628194s" totalImagesPullingTime="21.148365297s" totalInitContainerRuntime="3.715690737s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:00:50.240313527 +0000 UTC m=+885.113957151" watchObservedRunningTime="2026-08-01 00:00:50.241628194 +0000 UTC m=+885.115271804"
Aug 01 00:00:51.224860 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:51.224817    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 01 00:00:56.230049 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:56.230020    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:00:56.230621 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:00:56.230594    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 01 00:01:05.549388 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:05.549355    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:01:05.550453 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:05.550431    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:01:06.231383 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:06.231348    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 01 00:01:16.231174 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:16.231132    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 01 00:01:20.320286 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.320258    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-n7rff_b9d7aa3d-9b73-459a-a83b-8f3963caa28b/kserve-container/0.log"
Aug 01 00:01:20.544595 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.544553    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"]
Aug 01 00:01:20.544878 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.544844    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff" podUID="b9d7aa3d-9b73-459a-a83b-8f3963caa28b" containerName="kserve-container" containerID="cri-o://329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79" gracePeriod=30
Aug 01 00:01:20.545058 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.544883    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff" podUID="b9d7aa3d-9b73-459a-a83b-8f3963caa28b" containerName="kube-rbac-proxy" containerID="cri-o://81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc" gracePeriod=30
Aug 01 00:01:20.588656 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.588585    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"]
Aug 01 00:01:20.646778 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.646753    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"]
Aug 01 00:01:20.646889 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.646867    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:20.649666 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.649639    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Aug 01 00:01:20.649666 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.649651    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Aug 01 00:01:20.735803 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.735776    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2d8z\" (UniqueName: \"kubernetes.io/projected/65f37b52-81e7-477d-9ab0-d4946e959a08-kube-api-access-c2d8z\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:20.735939 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.735827    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65f37b52-81e7-477d-9ab0-d4946e959a08-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:20.735939 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.735914    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65f37b52-81e7-477d-9ab0-d4946e959a08-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:20.736047 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.735937    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65f37b52-81e7-477d-9ab0-d4946e959a08-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:20.736047 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.735996    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f37b52-81e7-477d-9ab0-d4946e959a08-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:20.822884 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.822858    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Aug 01 00:01:20.836777 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.836752    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65f37b52-81e7-477d-9ab0-d4946e959a08-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:20.836874 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.836782    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65f37b52-81e7-477d-9ab0-d4946e959a08-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:20.836874 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.836812    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f37b52-81e7-477d-9ab0-d4946e959a08-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:20.836874 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.836857    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2d8z\" (UniqueName: \"kubernetes.io/projected/65f37b52-81e7-477d-9ab0-d4946e959a08-kube-api-access-c2d8z\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:20.837022 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.836908    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65f37b52-81e7-477d-9ab0-d4946e959a08-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:20.837022 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:01:20.837005    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Aug 01 00:01:20.837127 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:01:20.837082    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f37b52-81e7-477d-9ab0-d4946e959a08-proxy-tls podName:65f37b52-81e7-477d-9ab0-d4946e959a08 nodeName:}" failed. No retries permitted until 2026-08-01 00:01:21.337062806 +0000 UTC m=+916.210706412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/65f37b52-81e7-477d-9ab0-d4946e959a08-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-dxv64" (UID: "65f37b52-81e7-477d-9ab0-d4946e959a08") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Aug 01 00:01:20.837193 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.837120    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65f37b52-81e7-477d-9ab0-d4946e959a08-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:20.837451 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.837431    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65f37b52-81e7-477d-9ab0-d4946e959a08-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:20.837581 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.837563    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65f37b52-81e7-477d-9ab0-d4946e959a08-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:20.846679 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.846625    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2d8z\" (UniqueName: \"kubernetes.io/projected/65f37b52-81e7-477d-9ab0-d4946e959a08-kube-api-access-c2d8z\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:20.937899 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.937866    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-proxy-tls\") pod \"b9d7aa3d-9b73-459a-a83b-8f3963caa28b\" (UID: \"b9d7aa3d-9b73-459a-a83b-8f3963caa28b\") "
Aug 01 00:01:20.938074 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.937924    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-kube-rbac-proxy-sar-config\") pod \"b9d7aa3d-9b73-459a-a83b-8f3963caa28b\" (UID: \"b9d7aa3d-9b73-459a-a83b-8f3963caa28b\") "
Aug 01 00:01:20.938074 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.938003    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwgm9\" (UniqueName: \"kubernetes.io/projected/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-kube-api-access-kwgm9\") pod \"b9d7aa3d-9b73-459a-a83b-8f3963caa28b\" (UID: \"b9d7aa3d-9b73-459a-a83b-8f3963caa28b\") "
Aug 01 00:01:20.938301 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.938276    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b9d7aa3d-9b73-459a-a83b-8f3963caa28b" (UID: "b9d7aa3d-9b73-459a-a83b-8f3963caa28b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:01:20.939977 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.939941    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b9d7aa3d-9b73-459a-a83b-8f3963caa28b" (UID: "b9d7aa3d-9b73-459a-a83b-8f3963caa28b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:01:20.940083 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:20.939992    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-kube-api-access-kwgm9" (OuterVolumeSpecName: "kube-api-access-kwgm9") pod "b9d7aa3d-9b73-459a-a83b-8f3963caa28b" (UID: "b9d7aa3d-9b73-459a-a83b-8f3963caa28b"). InnerVolumeSpecName "kube-api-access-kwgm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:01:21.038756 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.038711    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:01:21.038756 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.038750    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:01:21.038756 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.038760    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwgm9\" (UniqueName: \"kubernetes.io/projected/b9d7aa3d-9b73-459a-a83b-8f3963caa28b-kube-api-access-kwgm9\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:01:21.341125 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.341087    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f37b52-81e7-477d-9ab0-d4946e959a08-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:21.343400 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.343381    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f37b52-81e7-477d-9ab0-d4946e959a08-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dxv64\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:21.355109 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.355081    2584 generic.go:356] "Generic (PLEG): container finished" podID="b9d7aa3d-9b73-459a-a83b-8f3963caa28b" containerID="81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc" exitCode=2
Aug 01 00:01:21.355109 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.355111    2584 generic.go:356] "Generic (PLEG): container finished" podID="b9d7aa3d-9b73-459a-a83b-8f3963caa28b" containerID="329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79" exitCode=2
Aug 01 00:01:21.355264 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.355138    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"
Aug 01 00:01:21.355264 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.355167    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff" event={"ID":"b9d7aa3d-9b73-459a-a83b-8f3963caa28b","Type":"ContainerDied","Data":"81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc"}
Aug 01 00:01:21.355264 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.355227    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff" event={"ID":"b9d7aa3d-9b73-459a-a83b-8f3963caa28b","Type":"ContainerDied","Data":"329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79"}
Aug 01 00:01:21.355264 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.355249    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff" event={"ID":"b9d7aa3d-9b73-459a-a83b-8f3963caa28b","Type":"ContainerDied","Data":"62008c43334ce94ed8cc7e043b6061557e54b93bae1c01ce1e2539882dd2a49f"}
Aug 01 00:01:21.355396 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.355274    2584 scope.go:117] "RemoveContainer" containerID="81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc"
Aug 01 00:01:21.364131 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.364115    2584 scope.go:117] "RemoveContainer" containerID="329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79"
Aug 01 00:01:21.371471 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.371453    2584 scope.go:117] "RemoveContainer" containerID="81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc"
Aug 01 00:01:21.371722 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:01:21.371702    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc\": container with ID starting with 81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc not found: ID does not exist" containerID="81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc"
Aug 01 00:01:21.371775 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.371730    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc"} err="failed to get container status \"81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc\": rpc error: code = NotFound desc = could not find container \"81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc\": container with ID starting with 81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc not found: ID does not exist"
Aug 01 00:01:21.371775 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.371747    2584 scope.go:117] "RemoveContainer" containerID="329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79"
Aug 01 00:01:21.371989 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:01:21.371953    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79\": container with ID starting with 329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79 not found: ID does not exist" containerID="329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79"
Aug 01 00:01:21.372047 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.372001    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79"} err="failed to get container status \"329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79\": rpc error: code = NotFound desc = could not find container \"329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79\": container with ID starting with 329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79 not found: ID does not exist"
Aug 01 00:01:21.372047 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.372021    2584 scope.go:117] "RemoveContainer" containerID="81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc"
Aug 01 00:01:21.372221 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.372198    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc"} err="failed to get container status \"81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc\": rpc error: code = NotFound desc = could not find container \"81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc\": container with ID starting with 81f71816583aa72cf050a33eb68abed3030786882297e9bc09d0acf7926506fc not found: ID does not exist"
Aug 01 00:01:21.372271 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.372222    2584 scope.go:117] "RemoveContainer" containerID="329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79"
Aug 01 00:01:21.372411 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.372397    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79"} err="failed to get container status \"329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79\": rpc error: code = NotFound desc = could not find container \"329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79\": container with ID starting with 329cc98c5ac759eed5bcae9a4a9d74522481f6f4ec1957c4d2beae86f6188a79 not found: ID does not exist"
Aug 01 00:01:21.377854 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.377832    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"]
Aug 01 00:01:21.382102 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.382082    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-n7rff"]
Aug 01 00:01:21.571091 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.571054    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d7aa3d-9b73-459a-a83b-8f3963caa28b" path="/var/lib/kubelet/pods/b9d7aa3d-9b73-459a-a83b-8f3963caa28b/volumes"
Aug 01 00:01:21.611864 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.611804    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:01:21.758665 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:21.758636    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"]
Aug 01 00:01:21.760039 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:01:21.760014    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f37b52_81e7_477d_9ab0_d4946e959a08.slice/crio-b26ace3b974a64285dde325d220fee04bdebf000dad68be739b211e1663ae879 WatchSource:0}: Error finding container b26ace3b974a64285dde325d220fee04bdebf000dad68be739b211e1663ae879: Status 404 returned error can't find the container with id b26ace3b974a64285dde325d220fee04bdebf000dad68be739b211e1663ae879
Aug 01 00:01:22.363226 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:22.363188    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" event={"ID":"65f37b52-81e7-477d-9ab0-d4946e959a08","Type":"ContainerStarted","Data":"5835393ca827232593fb38e61d85f455b3dd84f3a6c04ad9353f4713dc6b5b80"}
Aug 01 00:01:22.363619 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:22.363255    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" event={"ID":"65f37b52-81e7-477d-9ab0-d4946e959a08","Type":"ContainerStarted","Data":"b26ace3b974a64285dde325d220fee04bdebf000dad68be739b211e1663ae879"}
Aug 01 00:01:26.231417 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:26.231381    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 01 00:01:36.231116 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:36.231076    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 01 00:01:42.450978 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:42.450879    2584 generic.go:356] "Generic (PLEG): container finished" podID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerID="5835393ca827232593fb38e61d85f455b3dd84f3a6c04ad9353f4713dc6b5b80" exitCode=0
Aug 01 00:01:42.450978 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:42.450952    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" event={"ID":"65f37b52-81e7-477d-9ab0-d4946e959a08","Type":"ContainerDied","Data":"5835393ca827232593fb38e61d85f455b3dd84f3a6c04ad9353f4713dc6b5b80"}
Aug 01 00:01:46.230604 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:46.230562    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 01 00:01:56.230630 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:56.230584    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 01 00:01:56.565866 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:01:56.565807    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 01 00:02:06.566196 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:06.566164    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:02:13.100564 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:13.100530    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"]
Aug 01 00:02:13.101055 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:13.100805    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kserve-container" containerID="cri-o://959e3bea7e5ce537c2719ea50f6ed36091ba5c6fd9e8bb89aeed234d984ca523" gracePeriod=30
Aug 01 00:02:13.101055 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:13.100835    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kube-rbac-proxy" containerID="cri-o://5f43bd2047fa5620b8cdf56550ceb9f4178a6832a3efabe43195ff7df486b416" gracePeriod=30
Aug 01 00:02:13.604376 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:13.604339    2584 generic.go:356] "Generic (PLEG): container finished" podID="44dadffc-1262-4294-b122-19e4da11a49d" containerID="5f43bd2047fa5620b8cdf56550ceb9f4178a6832a3efabe43195ff7df486b416" exitCode=2
Aug 01 00:02:13.604525 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:13.604424    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" event={"ID":"44dadffc-1262-4294-b122-19e4da11a49d","Type":"ContainerDied","Data":"5f43bd2047fa5620b8cdf56550ceb9f4178a6832a3efabe43195ff7df486b416"}
Aug 01 00:02:16.225372 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:16.225329    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Aug 01 00:02:16.565798 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:16.565750    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 01 00:02:18.193913 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.193845    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:02:18.330685 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.330650    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmxkb\" (UniqueName: \"kubernetes.io/projected/44dadffc-1262-4294-b122-19e4da11a49d-kube-api-access-xmxkb\") pod \"44dadffc-1262-4294-b122-19e4da11a49d\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") "
Aug 01 00:02:18.330872 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.330700    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44dadffc-1262-4294-b122-19e4da11a49d-kserve-provision-location\") pod \"44dadffc-1262-4294-b122-19e4da11a49d\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") "
Aug 01 00:02:18.330872 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.330734    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44dadffc-1262-4294-b122-19e4da11a49d-cabundle-cert\") pod \"44dadffc-1262-4294-b122-19e4da11a49d\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") "
Aug 01 00:02:18.330872 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.330772    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44dadffc-1262-4294-b122-19e4da11a49d-kube-rbac-proxy-sar-config\") pod \"44dadffc-1262-4294-b122-19e4da11a49d\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") "
Aug 01 00:02:18.330872 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.330830    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44dadffc-1262-4294-b122-19e4da11a49d-proxy-tls\") pod \"44dadffc-1262-4294-b122-19e4da11a49d\" (UID: \"44dadffc-1262-4294-b122-19e4da11a49d\") "
Aug 01 00:02:18.331152 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.331083    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44dadffc-1262-4294-b122-19e4da11a49d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "44dadffc-1262-4294-b122-19e4da11a49d" (UID: "44dadffc-1262-4294-b122-19e4da11a49d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:02:18.331594 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.331425    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44dadffc-1262-4294-b122-19e4da11a49d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "44dadffc-1262-4294-b122-19e4da11a49d" (UID: "44dadffc-1262-4294-b122-19e4da11a49d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:02:18.331594 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.331470    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44dadffc-1262-4294-b122-19e4da11a49d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "44dadffc-1262-4294-b122-19e4da11a49d" (UID: "44dadffc-1262-4294-b122-19e4da11a49d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:02:18.331594 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.331546    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44dadffc-1262-4294-b122-19e4da11a49d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:02:18.331594 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.331566    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44dadffc-1262-4294-b122-19e4da11a49d-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:02:18.333123 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.333086    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44dadffc-1262-4294-b122-19e4da11a49d-kube-api-access-xmxkb" (OuterVolumeSpecName: "kube-api-access-xmxkb") pod "44dadffc-1262-4294-b122-19e4da11a49d" (UID: "44dadffc-1262-4294-b122-19e4da11a49d"). InnerVolumeSpecName "kube-api-access-xmxkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:02:18.333325 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.333297    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44dadffc-1262-4294-b122-19e4da11a49d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44dadffc-1262-4294-b122-19e4da11a49d" (UID: "44dadffc-1262-4294-b122-19e4da11a49d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:02:18.432198 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.432164    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmxkb\" (UniqueName: \"kubernetes.io/projected/44dadffc-1262-4294-b122-19e4da11a49d-kube-api-access-xmxkb\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:02:18.432198 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.432200    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44dadffc-1262-4294-b122-19e4da11a49d-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:02:18.432442 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.432221    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44dadffc-1262-4294-b122-19e4da11a49d-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:02:18.631007 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.630958    2584 generic.go:356] "Generic (PLEG): container finished" podID="44dadffc-1262-4294-b122-19e4da11a49d" containerID="959e3bea7e5ce537c2719ea50f6ed36091ba5c6fd9e8bb89aeed234d984ca523" exitCode=0
Aug 01 00:02:18.631264 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.631061    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"
Aug 01 00:02:18.631264 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.631069    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" event={"ID":"44dadffc-1262-4294-b122-19e4da11a49d","Type":"ContainerDied","Data":"959e3bea7e5ce537c2719ea50f6ed36091ba5c6fd9e8bb89aeed234d984ca523"}
Aug 01 00:02:18.631264 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.631141    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv" event={"ID":"44dadffc-1262-4294-b122-19e4da11a49d","Type":"ContainerDied","Data":"9e32c4793027d4b7e80ddbb5942649366abf1a5be755628aa2632ed9eaee69c4"}
Aug 01 00:02:18.631264 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.631183    2584 scope.go:117] "RemoveContainer" containerID="5f43bd2047fa5620b8cdf56550ceb9f4178a6832a3efabe43195ff7df486b416"
Aug 01 00:02:18.641605 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.641482    2584 scope.go:117] "RemoveContainer" containerID="959e3bea7e5ce537c2719ea50f6ed36091ba5c6fd9e8bb89aeed234d984ca523"
Aug 01 00:02:18.649941 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.649925    2584 scope.go:117] "RemoveContainer" containerID="2bd54f1d5a2e3ae775de838100f1e733ea974a3d28ab48c04b207b38b48ea7d3"
Aug 01 00:02:18.656791 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.656772    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"]
Aug 01 00:02:18.659018 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.658997    2584 scope.go:117] "RemoveContainer" containerID="5f43bd2047fa5620b8cdf56550ceb9f4178a6832a3efabe43195ff7df486b416"
Aug 01 00:02:18.659311 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:02:18.659289    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f43bd2047fa5620b8cdf56550ceb9f4178a6832a3efabe43195ff7df486b416\": container with ID starting with 5f43bd2047fa5620b8cdf56550ceb9f4178a6832a3efabe43195ff7df486b416 not found: ID does not exist" containerID="5f43bd2047fa5620b8cdf56550ceb9f4178a6832a3efabe43195ff7df486b416"
Aug 01 00:02:18.659404 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.659321    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f43bd2047fa5620b8cdf56550ceb9f4178a6832a3efabe43195ff7df486b416"} err="failed to get container status \"5f43bd2047fa5620b8cdf56550ceb9f4178a6832a3efabe43195ff7df486b416\": rpc error: code = NotFound desc = could not find container \"5f43bd2047fa5620b8cdf56550ceb9f4178a6832a3efabe43195ff7df486b416\": container with ID starting with 5f43bd2047fa5620b8cdf56550ceb9f4178a6832a3efabe43195ff7df486b416 not found: ID does not exist"
Aug 01 00:02:18.659404 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.659345    2584 scope.go:117] "RemoveContainer" containerID="959e3bea7e5ce537c2719ea50f6ed36091ba5c6fd9e8bb89aeed234d984ca523"
Aug 01 00:02:18.659675 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:02:18.659651    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959e3bea7e5ce537c2719ea50f6ed36091ba5c6fd9e8bb89aeed234d984ca523\": container with ID starting with 959e3bea7e5ce537c2719ea50f6ed36091ba5c6fd9e8bb89aeed234d984ca523 not found: ID does not exist" containerID="959e3bea7e5ce537c2719ea50f6ed36091ba5c6fd9e8bb89aeed234d984ca523"
Aug 01 00:02:18.659748 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.659684    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959e3bea7e5ce537c2719ea50f6ed36091ba5c6fd9e8bb89aeed234d984ca523"} err="failed to get container status \"959e3bea7e5ce537c2719ea50f6ed36091ba5c6fd9e8bb89aeed234d984ca523\": rpc error: code = NotFound desc = could not find container \"959e3bea7e5ce537c2719ea50f6ed36091ba5c6fd9e8bb89aeed234d984ca523\": container with ID starting with 959e3bea7e5ce537c2719ea50f6ed36091ba5c6fd9e8bb89aeed234d984ca523 not found: ID does not exist"
Aug 01 00:02:18.659748 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.659708    2584 scope.go:117] "RemoveContainer" containerID="2bd54f1d5a2e3ae775de838100f1e733ea974a3d28ab48c04b207b38b48ea7d3"
Aug 01 00:02:18.659998 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:02:18.659957    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd54f1d5a2e3ae775de838100f1e733ea974a3d28ab48c04b207b38b48ea7d3\": container with ID starting with 2bd54f1d5a2e3ae775de838100f1e733ea974a3d28ab48c04b207b38b48ea7d3 not found: ID does not exist" containerID="2bd54f1d5a2e3ae775de838100f1e733ea974a3d28ab48c04b207b38b48ea7d3"
Aug 01 00:02:18.660105 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.660000    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd54f1d5a2e3ae775de838100f1e733ea974a3d28ab48c04b207b38b48ea7d3"} err="failed to get container status \"2bd54f1d5a2e3ae775de838100f1e733ea974a3d28ab48c04b207b38b48ea7d3\": rpc error: code = NotFound desc = could not find container \"2bd54f1d5a2e3ae775de838100f1e733ea974a3d28ab48c04b207b38b48ea7d3\": container with ID starting with 2bd54f1d5a2e3ae775de838100f1e733ea974a3d28ab48c04b207b38b48ea7d3 not found: ID does not exist"
Aug 01 00:02:18.660635 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:18.660617    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-tc7qv"]
Aug 01 00:02:19.571811 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:02:19.571767    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44dadffc-1262-4294-b122-19e4da11a49d" path="/var/lib/kubelet/pods/44dadffc-1262-4294-b122-19e4da11a49d/volumes"
Aug 01 00:03:40.022863 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:03:40.022818    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" event={"ID":"65f37b52-81e7-477d-9ab0-d4946e959a08","Type":"ContainerStarted","Data":"9dc4253e84b991f963970e7c80e61bb0f95df49155b400ca76a15b31cbf67ed8"}
Aug 01 00:03:40.023290 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:03:40.022892    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" event={"ID":"65f37b52-81e7-477d-9ab0-d4946e959a08","Type":"ContainerStarted","Data":"7e5778e9b7efdda203fee2467100a2c0df0047498cb01ec32f809bef758e8c93"}
Aug 01 00:03:40.023290 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:03:40.023034    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:03:40.042046 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:03:40.042005    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" podStartSLOduration=3.106070722 podStartE2EDuration="2m20.041991331s" totalImagesPullingTime="1m56.766303585s" totalInitContainerRuntime="20.169617024s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:03:40.040988125 +0000 UTC m=+1054.914631752" watchObservedRunningTime="2026-08-01 00:03:40.041991331 +0000 UTC m=+1054.915634932"
Aug 01 00:03:41.027335 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:03:41.027297    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:03:41.028644 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:03:41.028621    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 01 00:03:42.031216 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:03:42.031172    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 01 00:03:47.036455 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:03:47.036425    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:03:47.037026 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:03:47.037000    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 01 00:03:57.038134 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:03:57.038105    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:04:06.846291 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:06.846211    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"]
Aug 01 00:04:06.846811 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:06.846427    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerName="kserve-container" containerID="cri-o://7e5778e9b7efdda203fee2467100a2c0df0047498cb01ec32f809bef758e8c93" gracePeriod=30
Aug 01 00:04:06.846811 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:06.846478    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerName="kube-rbac-proxy" containerID="cri-o://9dc4253e84b991f963970e7c80e61bb0f95df49155b400ca76a15b31cbf67ed8" gracePeriod=30
Aug 01 00:04:07.031593 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:07.031540    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Aug 01 00:04:07.037272 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:07.037234    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 01 00:04:07.138994 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:07.138899    2584 generic.go:356] "Generic (PLEG): container finished" podID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerID="9dc4253e84b991f963970e7c80e61bb0f95df49155b400ca76a15b31cbf67ed8" exitCode=2
Aug 01 00:04:07.139157 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:07.138992    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" event={"ID":"65f37b52-81e7-477d-9ab0-d4946e959a08","Type":"ContainerDied","Data":"9dc4253e84b991f963970e7c80e61bb0f95df49155b400ca76a15b31cbf67ed8"}
Aug 01 00:04:08.152114 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.152077    2584 generic.go:356] "Generic (PLEG): container finished" podID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerID="7e5778e9b7efdda203fee2467100a2c0df0047498cb01ec32f809bef758e8c93" exitCode=0
Aug 01 00:04:08.152563 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.152096    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" event={"ID":"65f37b52-81e7-477d-9ab0-d4946e959a08","Type":"ContainerDied","Data":"7e5778e9b7efdda203fee2467100a2c0df0047498cb01ec32f809bef758e8c93"}
Aug 01 00:04:08.237742 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.237717    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:04:08.368757 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.368685    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f37b52-81e7-477d-9ab0-d4946e959a08-proxy-tls\") pod \"65f37b52-81e7-477d-9ab0-d4946e959a08\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") "
Aug 01 00:04:08.368757 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.368726    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65f37b52-81e7-477d-9ab0-d4946e959a08-kserve-provision-location\") pod \"65f37b52-81e7-477d-9ab0-d4946e959a08\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") "
Aug 01 00:04:08.368757 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.368744    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65f37b52-81e7-477d-9ab0-d4946e959a08-cabundle-cert\") pod \"65f37b52-81e7-477d-9ab0-d4946e959a08\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") "
Aug 01 00:04:08.369130 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.368797    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65f37b52-81e7-477d-9ab0-d4946e959a08-kube-rbac-proxy-sar-config\") pod \"65f37b52-81e7-477d-9ab0-d4946e959a08\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") "
Aug 01 00:04:08.369130 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.368821    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2d8z\" (UniqueName: \"kubernetes.io/projected/65f37b52-81e7-477d-9ab0-d4946e959a08-kube-api-access-c2d8z\") pod \"65f37b52-81e7-477d-9ab0-d4946e959a08\" (UID: \"65f37b52-81e7-477d-9ab0-d4946e959a08\") "
Aug 01 00:04:08.369240 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.369150    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f37b52-81e7-477d-9ab0-d4946e959a08-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "65f37b52-81e7-477d-9ab0-d4946e959a08" (UID: "65f37b52-81e7-477d-9ab0-d4946e959a08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:04:08.369240 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.369165    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f37b52-81e7-477d-9ab0-d4946e959a08-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "65f37b52-81e7-477d-9ab0-d4946e959a08" (UID: "65f37b52-81e7-477d-9ab0-d4946e959a08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:04:08.370902 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.370873    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f37b52-81e7-477d-9ab0-d4946e959a08-kube-api-access-c2d8z" (OuterVolumeSpecName: "kube-api-access-c2d8z") pod "65f37b52-81e7-477d-9ab0-d4946e959a08" (UID: "65f37b52-81e7-477d-9ab0-d4946e959a08"). InnerVolumeSpecName "kube-api-access-c2d8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:04:08.371020 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.370881    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f37b52-81e7-477d-9ab0-d4946e959a08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65f37b52-81e7-477d-9ab0-d4946e959a08" (UID: "65f37b52-81e7-477d-9ab0-d4946e959a08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:04:08.373522 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.373493    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f37b52-81e7-477d-9ab0-d4946e959a08-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "65f37b52-81e7-477d-9ab0-d4946e959a08" (UID: "65f37b52-81e7-477d-9ab0-d4946e959a08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:04:08.470084 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.470056    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65f37b52-81e7-477d-9ab0-d4946e959a08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:04:08.470084 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.470080    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2d8z\" (UniqueName: \"kubernetes.io/projected/65f37b52-81e7-477d-9ab0-d4946e959a08-kube-api-access-c2d8z\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:04:08.470251 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.470097    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f37b52-81e7-477d-9ab0-d4946e959a08-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:04:08.470251 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.470107    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65f37b52-81e7-477d-9ab0-d4946e959a08-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:04:08.470251 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:08.470116    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65f37b52-81e7-477d-9ab0-d4946e959a08-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:04:09.157939 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:09.157904    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64" event={"ID":"65f37b52-81e7-477d-9ab0-d4946e959a08","Type":"ContainerDied","Data":"b26ace3b974a64285dde325d220fee04bdebf000dad68be739b211e1663ae879"}
Aug 01 00:04:09.158433 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:09.157914    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"
Aug 01 00:04:09.158433 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:09.157987    2584 scope.go:117] "RemoveContainer" containerID="9dc4253e84b991f963970e7c80e61bb0f95df49155b400ca76a15b31cbf67ed8"
Aug 01 00:04:09.166834 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:09.166813    2584 scope.go:117] "RemoveContainer" containerID="7e5778e9b7efdda203fee2467100a2c0df0047498cb01ec32f809bef758e8c93"
Aug 01 00:04:09.173873 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:09.173856    2584 scope.go:117] "RemoveContainer" containerID="5835393ca827232593fb38e61d85f455b3dd84f3a6c04ad9353f4713dc6b5b80"
Aug 01 00:04:09.181419 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:09.181399    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"]
Aug 01 00:04:09.185141 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:09.185122    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dxv64"]
Aug 01 00:04:09.570943 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:09.570910    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" path="/var/lib/kubelet/pods/65f37b52-81e7-477d-9ab0-d4946e959a08/volumes"
Aug 01 00:04:12.116560 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116521    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"]
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116814    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kube-rbac-proxy"
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116825    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kube-rbac-proxy"
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116838    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerName="kserve-container"
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116844    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerName="kserve-container"
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116853    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d7aa3d-9b73-459a-a83b-8f3963caa28b" containerName="kserve-container"
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116859    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d7aa3d-9b73-459a-a83b-8f3963caa28b" containerName="kserve-container"
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116869    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerName="storage-initializer"
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116874    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerName="storage-initializer"
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116886    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="storage-initializer"
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116891    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="storage-initializer"
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116899    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d7aa3d-9b73-459a-a83b-8f3963caa28b" containerName="kube-rbac-proxy"
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116905    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d7aa3d-9b73-459a-a83b-8f3963caa28b" containerName="kube-rbac-proxy"
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116910    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerName="kube-rbac-proxy"
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116915    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerName="kube-rbac-proxy"
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116922    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kserve-container"
Aug 01 00:04:12.116937 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116928    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kserve-container"
Aug 01 00:04:12.117497 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116987    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kube-rbac-proxy"
Aug 01 00:04:12.117497 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.116997    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d7aa3d-9b73-459a-a83b-8f3963caa28b" containerName="kube-rbac-proxy"
Aug 01 00:04:12.117497 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.117003    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerName="kube-rbac-proxy"
Aug 01 00:04:12.117497 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.117010    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d7aa3d-9b73-459a-a83b-8f3963caa28b" containerName="kserve-container"
Aug 01 00:04:12.117497 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.117015    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="44dadffc-1262-4294-b122-19e4da11a49d" containerName="kserve-container"
Aug 01 00:04:12.117497 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.117020    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f37b52-81e7-477d-9ab0-d4946e959a08" containerName="kserve-container"
Aug 01 00:04:12.136063 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.136031    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.136266 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.136221    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"]
Aug 01 00:04:12.141184 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.140729    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:04:12.141184 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.141044    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:04:12.141512 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.141493    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:04:12.141592 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.141508    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Aug 01 00:04:12.141592 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.141504    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Aug 01 00:04:12.141592 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.141556    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:04:12.141592 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.141504    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:04:12.300815 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.300783    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-75kp7\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.301076 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.300824    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-792w4\" (UniqueName: \"kubernetes.io/projected/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kube-api-access-792w4\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-75kp7\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.301076 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.300845    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-75kp7\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.301076 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.300872    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-75kp7\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.301076 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.300887    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-75kp7\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.402022 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.401902    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-792w4\" (UniqueName: \"kubernetes.io/projected/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kube-api-access-792w4\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-75kp7\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.402022 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.401934    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-75kp7\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.402022 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.401985    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-75kp7\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.402022 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.402002    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-75kp7\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.402349 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.402062    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-75kp7\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.402575 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.402554    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-75kp7\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.402754 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.402735    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-75kp7\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.402822 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.402739    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-75kp7\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.404546 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.404522    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-75kp7\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.412379 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.412346    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-792w4\" (UniqueName: \"kubernetes.io/projected/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kube-api-access-792w4\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-75kp7\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.447270 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.447245    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:12.594169 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:12.594129    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"]
Aug 01 00:04:12.596753 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:04:12.596723    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f1a158_6ddd_4ddd_9140_1f96bf32439f.slice/crio-b61716a3e454bbaa3a3f7a6ec045a67aa78c46f2d389808ee8ad237559e2326b WatchSource:0}: Error finding container b61716a3e454bbaa3a3f7a6ec045a67aa78c46f2d389808ee8ad237559e2326b: Status 404 returned error can't find the container with id b61716a3e454bbaa3a3f7a6ec045a67aa78c46f2d389808ee8ad237559e2326b
Aug 01 00:04:13.177480 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:13.177432    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7" event={"ID":"a9f1a158-6ddd-4ddd-9140-1f96bf32439f","Type":"ContainerStarted","Data":"683d9d614541e59062cb1684671bcd5e49e7b924ebfa09350c0987323e2b535c"}
Aug 01 00:04:13.177943 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:13.177509    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7" event={"ID":"a9f1a158-6ddd-4ddd-9140-1f96bf32439f","Type":"ContainerStarted","Data":"b61716a3e454bbaa3a3f7a6ec045a67aa78c46f2d389808ee8ad237559e2326b"}
Aug 01 00:04:43.593760 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:04:43.593726    2584 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f1a158_6ddd_4ddd_9140_1f96bf32439f.slice/crio-683d9d614541e59062cb1684671bcd5e49e7b924ebfa09350c0987323e2b535c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f1a158_6ddd_4ddd_9140_1f96bf32439f.slice/crio-conmon-683d9d614541e59062cb1684671bcd5e49e7b924ebfa09350c0987323e2b535c.scope\": RecentStats: unable to find data in memory cache]"
Aug 01 00:04:44.313494 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:44.313446    2584 generic.go:356] "Generic (PLEG): container finished" podID="a9f1a158-6ddd-4ddd-9140-1f96bf32439f" containerID="683d9d614541e59062cb1684671bcd5e49e7b924ebfa09350c0987323e2b535c" exitCode=0
Aug 01 00:04:44.313646 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:44.313520    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7" event={"ID":"a9f1a158-6ddd-4ddd-9140-1f96bf32439f","Type":"ContainerDied","Data":"683d9d614541e59062cb1684671bcd5e49e7b924ebfa09350c0987323e2b535c"}
Aug 01 00:04:45.320977 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:45.320929    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7" event={"ID":"a9f1a158-6ddd-4ddd-9140-1f96bf32439f","Type":"ContainerStarted","Data":"6d51b6080faf599817feb45ac25cd0b73e9545447c15f9069a05e4dc4eeef8c0"}
Aug 01 00:04:45.321424 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:45.321022    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7" event={"ID":"a9f1a158-6ddd-4ddd-9140-1f96bf32439f","Type":"ContainerStarted","Data":"1292daf5992d0320487f062b74c9e3df32641d4a25b1b255c0f8f7bb83a55c64"}
Aug 01 00:04:45.321424 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:45.321243    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:45.342241 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:45.342192    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7" podStartSLOduration=2.4874099100000002 podStartE2EDuration="33.342177466s" totalImagesPullingTime="0s" totalInitContainerRuntime="30.854767556s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:04:45.339761211 +0000 UTC m=+1120.213404860" watchObservedRunningTime="2026-08-01 00:04:45.342177466 +0000 UTC m=+1120.215821074"
Aug 01 00:04:46.325353 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:46.325329    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:04:52.335026 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:04:52.334999    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:06:05.582065 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:05.582036    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:06:05.583543 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:05.583518    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:06:19.571365 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:19.571337    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:06:28.312943 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:28.312899    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"]
Aug 01 00:06:28.313405 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:28.313308    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7" podUID="a9f1a158-6ddd-4ddd-9140-1f96bf32439f" containerName="kube-rbac-proxy" containerID="cri-o://6d51b6080faf599817feb45ac25cd0b73e9545447c15f9069a05e4dc4eeef8c0" gracePeriod=30
Aug 01 00:06:28.313405 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:28.313299    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7" podUID="a9f1a158-6ddd-4ddd-9140-1f96bf32439f" containerName="kserve-container" containerID="cri-o://1292daf5992d0320487f062b74c9e3df32641d4a25b1b255c0f8f7bb83a55c64" gracePeriod=30
Aug 01 00:06:28.768399 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:28.768309    2584 generic.go:356] "Generic (PLEG): container finished" podID="a9f1a158-6ddd-4ddd-9140-1f96bf32439f" containerID="6d51b6080faf599817feb45ac25cd0b73e9545447c15f9069a05e4dc4eeef8c0" exitCode=2
Aug 01 00:06:28.768399 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:28.768384    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7" event={"ID":"a9f1a158-6ddd-4ddd-9140-1f96bf32439f","Type":"ContainerDied","Data":"6d51b6080faf599817feb45ac25cd0b73e9545447c15f9069a05e4dc4eeef8c0"}
Aug 01 00:06:29.566467 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.566432    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7" podUID="a9f1a158-6ddd-4ddd-9140-1f96bf32439f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-autogluon-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 01 00:06:29.759793 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.759770    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:06:29.774117 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.774089    2584 generic.go:356] "Generic (PLEG): container finished" podID="a9f1a158-6ddd-4ddd-9140-1f96bf32439f" containerID="1292daf5992d0320487f062b74c9e3df32641d4a25b1b255c0f8f7bb83a55c64" exitCode=0
Aug 01 00:06:29.774274 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.774206    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7" event={"ID":"a9f1a158-6ddd-4ddd-9140-1f96bf32439f","Type":"ContainerDied","Data":"1292daf5992d0320487f062b74c9e3df32641d4a25b1b255c0f8f7bb83a55c64"}
Aug 01 00:06:29.774274 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.774265    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7" event={"ID":"a9f1a158-6ddd-4ddd-9140-1f96bf32439f","Type":"ContainerDied","Data":"b61716a3e454bbaa3a3f7a6ec045a67aa78c46f2d389808ee8ad237559e2326b"}
Aug 01 00:06:29.774389 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.774212    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"
Aug 01 00:06:29.774389 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.774334    2584 scope.go:117] "RemoveContainer" containerID="6d51b6080faf599817feb45ac25cd0b73e9545447c15f9069a05e4dc4eeef8c0"
Aug 01 00:06:29.782277 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.782251    2584 scope.go:117] "RemoveContainer" containerID="1292daf5992d0320487f062b74c9e3df32641d4a25b1b255c0f8f7bb83a55c64"
Aug 01 00:06:29.789549 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.789527    2584 scope.go:117] "RemoveContainer" containerID="683d9d614541e59062cb1684671bcd5e49e7b924ebfa09350c0987323e2b535c"
Aug 01 00:06:29.796627 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.796608    2584 scope.go:117] "RemoveContainer" containerID="6d51b6080faf599817feb45ac25cd0b73e9545447c15f9069a05e4dc4eeef8c0"
Aug 01 00:06:29.796933 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:06:29.796913    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d51b6080faf599817feb45ac25cd0b73e9545447c15f9069a05e4dc4eeef8c0\": container with ID starting with 6d51b6080faf599817feb45ac25cd0b73e9545447c15f9069a05e4dc4eeef8c0 not found: ID does not exist" containerID="6d51b6080faf599817feb45ac25cd0b73e9545447c15f9069a05e4dc4eeef8c0"
Aug 01 00:06:29.797049 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.796945    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d51b6080faf599817feb45ac25cd0b73e9545447c15f9069a05e4dc4eeef8c0"} err="failed to get container status \"6d51b6080faf599817feb45ac25cd0b73e9545447c15f9069a05e4dc4eeef8c0\": rpc error: code = NotFound desc = could not find container \"6d51b6080faf599817feb45ac25cd0b73e9545447c15f9069a05e4dc4eeef8c0\": container with ID starting with 6d51b6080faf599817feb45ac25cd0b73e9545447c15f9069a05e4dc4eeef8c0 not found: ID does not exist"
Aug 01 00:06:29.797049 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.796987    2584 scope.go:117] "RemoveContainer" containerID="1292daf5992d0320487f062b74c9e3df32641d4a25b1b255c0f8f7bb83a55c64"
Aug 01 00:06:29.797345 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:06:29.797323    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1292daf5992d0320487f062b74c9e3df32641d4a25b1b255c0f8f7bb83a55c64\": container with ID starting with 1292daf5992d0320487f062b74c9e3df32641d4a25b1b255c0f8f7bb83a55c64 not found: ID does not exist" containerID="1292daf5992d0320487f062b74c9e3df32641d4a25b1b255c0f8f7bb83a55c64"
Aug 01 00:06:29.797411 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.797350    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1292daf5992d0320487f062b74c9e3df32641d4a25b1b255c0f8f7bb83a55c64"} err="failed to get container status \"1292daf5992d0320487f062b74c9e3df32641d4a25b1b255c0f8f7bb83a55c64\": rpc error: code = NotFound desc = could not find container \"1292daf5992d0320487f062b74c9e3df32641d4a25b1b255c0f8f7bb83a55c64\": container with ID starting with 1292daf5992d0320487f062b74c9e3df32641d4a25b1b255c0f8f7bb83a55c64 not found: ID does not exist"
Aug 01 00:06:29.797411 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.797366    2584 scope.go:117] "RemoveContainer" containerID="683d9d614541e59062cb1684671bcd5e49e7b924ebfa09350c0987323e2b535c"
Aug 01 00:06:29.797954 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:06:29.797833    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683d9d614541e59062cb1684671bcd5e49e7b924ebfa09350c0987323e2b535c\": container with ID starting with 683d9d614541e59062cb1684671bcd5e49e7b924ebfa09350c0987323e2b535c not found: ID does not exist" containerID="683d9d614541e59062cb1684671bcd5e49e7b924ebfa09350c0987323e2b535c"
Aug 01 00:06:29.797954 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.797866    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683d9d614541e59062cb1684671bcd5e49e7b924ebfa09350c0987323e2b535c"} err="failed to get container status \"683d9d614541e59062cb1684671bcd5e49e7b924ebfa09350c0987323e2b535c\": rpc error: code = NotFound desc = could not find container \"683d9d614541e59062cb1684671bcd5e49e7b924ebfa09350c0987323e2b535c\": container with ID starting with 683d9d614541e59062cb1684671bcd5e49e7b924ebfa09350c0987323e2b535c not found: ID does not exist"
Aug 01 00:06:29.849158 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.849127    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kube-rbac-proxy-sar-config\") pod \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") "
Aug 01 00:06:29.849325 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.849174    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-792w4\" (UniqueName: \"kubernetes.io/projected/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kube-api-access-792w4\") pod \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") "
Aug 01 00:06:29.849325 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.849199    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kserve-provision-location\") pod \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") "
Aug 01 00:06:29.849325 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.849259    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-cabundle-cert\") pod \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") "
Aug 01 00:06:29.849325 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.849280    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-proxy-tls\") pod \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\" (UID: \"a9f1a158-6ddd-4ddd-9140-1f96bf32439f\") "
Aug 01 00:06:29.849566 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.849540    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a9f1a158-6ddd-4ddd-9140-1f96bf32439f" (UID: "a9f1a158-6ddd-4ddd-9140-1f96bf32439f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:06:29.849635 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.849569    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a9f1a158-6ddd-4ddd-9140-1f96bf32439f" (UID: "a9f1a158-6ddd-4ddd-9140-1f96bf32439f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:06:29.851327 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.851299    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kube-api-access-792w4" (OuterVolumeSpecName: "kube-api-access-792w4") pod "a9f1a158-6ddd-4ddd-9140-1f96bf32439f" (UID: "a9f1a158-6ddd-4ddd-9140-1f96bf32439f"). InnerVolumeSpecName "kube-api-access-792w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:06:29.851456 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.851394    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9f1a158-6ddd-4ddd-9140-1f96bf32439f" (UID: "a9f1a158-6ddd-4ddd-9140-1f96bf32439f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:06:29.854103 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.854083    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a9f1a158-6ddd-4ddd-9140-1f96bf32439f" (UID: "a9f1a158-6ddd-4ddd-9140-1f96bf32439f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:06:29.950676 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.950639    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:06:29.950676 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.950669    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:06:29.950676 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.950679    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:06:29.950896 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.950689    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-792w4\" (UniqueName: \"kubernetes.io/projected/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kube-api-access-792w4\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:06:29.950896 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:29.950698    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9f1a158-6ddd-4ddd-9140-1f96bf32439f-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:06:30.100543 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:30.100510    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"]
Aug 01 00:06:30.104421 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:30.104398    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-75kp7"]
Aug 01 00:06:31.570415 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:06:31.570380    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9f1a158-6ddd-4ddd-9140-1f96bf32439f" path="/var/lib/kubelet/pods/a9f1a158-6ddd-4ddd-9140-1f96bf32439f/volumes"
Aug 01 00:07:25.031437 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.031401    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"]
Aug 01 00:07:25.032042 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.031694    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f1a158-6ddd-4ddd-9140-1f96bf32439f" containerName="kube-rbac-proxy"
Aug 01 00:07:25.032042 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.031705    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f1a158-6ddd-4ddd-9140-1f96bf32439f" containerName="kube-rbac-proxy"
Aug 01 00:07:25.032042 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.031719    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f1a158-6ddd-4ddd-9140-1f96bf32439f" containerName="kserve-container"
Aug 01 00:07:25.032042 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.031725    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f1a158-6ddd-4ddd-9140-1f96bf32439f" containerName="kserve-container"
Aug 01 00:07:25.032042 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.031738    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f1a158-6ddd-4ddd-9140-1f96bf32439f" containerName="storage-initializer"
Aug 01 00:07:25.032042 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.031744    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f1a158-6ddd-4ddd-9140-1f96bf32439f" containerName="storage-initializer"
Aug 01 00:07:25.032042 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.031788    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f1a158-6ddd-4ddd-9140-1f96bf32439f" containerName="kserve-container"
Aug 01 00:07:25.032042 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.031795    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f1a158-6ddd-4ddd-9140-1f96bf32439f" containerName="kube-rbac-proxy"
Aug 01 00:07:25.034040 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.034021    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.036732 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.036705    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:07:25.036732 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.036721    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Aug 01 00:07:25.036923 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.036725    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:07:25.036923 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.036785    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:07:25.036923 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.036711    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Aug 01 00:07:25.037170 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.037150    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:07:25.037612 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.037600    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:07:25.045621 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.045599    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"]
Aug 01 00:07:25.197999 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.197953    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221cc730-d5c4-459b-87ed-877793f701fd-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.198195 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.198034    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/221cc730-d5c4-459b-87ed-877793f701fd-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.198195 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.198063    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9kgh\" (UniqueName: \"kubernetes.io/projected/221cc730-d5c4-459b-87ed-877793f701fd-kube-api-access-t9kgh\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.198195 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.198130    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/221cc730-d5c4-459b-87ed-877793f701fd-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.198319 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.198199    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/221cc730-d5c4-459b-87ed-877793f701fd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.298874 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.298840    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/221cc730-d5c4-459b-87ed-877793f701fd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.299062 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.298885    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221cc730-d5c4-459b-87ed-877793f701fd-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.299062 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.298937    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/221cc730-d5c4-459b-87ed-877793f701fd-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.299062 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.298992    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9kgh\" (UniqueName: \"kubernetes.io/projected/221cc730-d5c4-459b-87ed-877793f701fd-kube-api-access-t9kgh\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.299062 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.299036    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/221cc730-d5c4-459b-87ed-877793f701fd-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.299290 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:07:25.299086    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Aug 01 00:07:25.299290 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:07:25.299153    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/221cc730-d5c4-459b-87ed-877793f701fd-proxy-tls podName:221cc730-d5c4-459b-87ed-877793f701fd nodeName:}" failed. No retries permitted until 2026-08-01 00:07:25.79913653 +0000 UTC m=+1280.672780122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/221cc730-d5c4-459b-87ed-877793f701fd-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" (UID: "221cc730-d5c4-459b-87ed-877793f701fd") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Aug 01 00:07:25.299449 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.299426    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/221cc730-d5c4-459b-87ed-877793f701fd-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.299722 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.299702    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/221cc730-d5c4-459b-87ed-877793f701fd-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.299762 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.299702    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/221cc730-d5c4-459b-87ed-877793f701fd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.307591 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.307567    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9kgh\" (UniqueName: \"kubernetes.io/projected/221cc730-d5c4-459b-87ed-877793f701fd-kube-api-access-t9kgh\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.802406 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.802376    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221cc730-d5c4-459b-87ed-877793f701fd-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.804909 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.804874    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221cc730-d5c4-459b-87ed-877793f701fd-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:25.945072 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:25.945032    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:26.075772 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:26.075688    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"]
Aug 01 00:07:26.079067 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:07:26.079042    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221cc730_d5c4_459b_87ed_877793f701fd.slice/crio-446bc572ada7dcb2ef00fde5ab4b16cb2b3ec5d7fd3b2f389979f735d9658b2a WatchSource:0}: Error finding container 446bc572ada7dcb2ef00fde5ab4b16cb2b3ec5d7fd3b2f389979f735d9658b2a: Status 404 returned error can't find the container with id 446bc572ada7dcb2ef00fde5ab4b16cb2b3ec5d7fd3b2f389979f735d9658b2a
Aug 01 00:07:27.021546 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:27.021506    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" event={"ID":"221cc730-d5c4-459b-87ed-877793f701fd","Type":"ContainerStarted","Data":"7909e4b5d137c4d16670c7459869abfd1117bfdb3560007bbfa95c0bf7b298f1"}
Aug 01 00:07:27.021712 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:27.021599    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" event={"ID":"221cc730-d5c4-459b-87ed-877793f701fd","Type":"ContainerStarted","Data":"446bc572ada7dcb2ef00fde5ab4b16cb2b3ec5d7fd3b2f389979f735d9658b2a"}
Aug 01 00:07:30.035646 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:30.035566    2584 generic.go:356] "Generic (PLEG): container finished" podID="221cc730-d5c4-459b-87ed-877793f701fd" containerID="7909e4b5d137c4d16670c7459869abfd1117bfdb3560007bbfa95c0bf7b298f1" exitCode=0
Aug 01 00:07:30.035646 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:30.035633    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" event={"ID":"221cc730-d5c4-459b-87ed-877793f701fd","Type":"ContainerDied","Data":"7909e4b5d137c4d16670c7459869abfd1117bfdb3560007bbfa95c0bf7b298f1"}
Aug 01 00:07:31.041784 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:31.041749    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" event={"ID":"221cc730-d5c4-459b-87ed-877793f701fd","Type":"ContainerStarted","Data":"d22665c5bb422493eb0cfc770d19261a8c306e92e2ca449c3969fae7ff2cda4f"}
Aug 01 00:07:31.042185 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:31.041814    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" event={"ID":"221cc730-d5c4-459b-87ed-877793f701fd","Type":"ContainerStarted","Data":"511d75acc839bf33a6fb3ae086b859b30842d582b4fea01d1ab3107e69e1435d"}
Aug 01 00:07:31.042185 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:31.042131    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:31.042262 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:31.042189    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:07:31.061219 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:31.061176    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" podStartSLOduration=2.545184155 podStartE2EDuration="6.061165277s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.515981122s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:07:31.060621432 +0000 UTC m=+1285.934265144" watchObservedRunningTime="2026-08-01 00:07:31.061165277 +0000 UTC m=+1285.934808887"
Aug 01 00:07:37.056940 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:07:37.056910    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:08:07.058163 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:07.058119    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 01 00:08:17.058251 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:17.058209    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 01 00:08:27.057576 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:27.057539    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 01 00:08:37.057727 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:37.057683    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 01 00:08:47.120879 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:47.120843    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:08:55.158709 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:55.158667    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"]
Aug 01 00:08:55.159404 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:55.158935    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="kserve-container" containerID="cri-o://511d75acc839bf33a6fb3ae086b859b30842d582b4fea01d1ab3107e69e1435d" gracePeriod=30
Aug 01 00:08:55.159404 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:55.159005    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="kube-rbac-proxy" containerID="cri-o://d22665c5bb422493eb0cfc770d19261a8c306e92e2ca449c3969fae7ff2cda4f" gracePeriod=30
Aug 01 00:08:55.406402 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:55.406370    2584 generic.go:356] "Generic (PLEG): container finished" podID="221cc730-d5c4-459b-87ed-877793f701fd" containerID="d22665c5bb422493eb0cfc770d19261a8c306e92e2ca449c3969fae7ff2cda4f" exitCode=2
Aug 01 00:08:55.406402 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:55.406406    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" event={"ID":"221cc730-d5c4-459b-87ed-877793f701fd","Type":"ContainerDied","Data":"d22665c5bb422493eb0cfc770d19261a8c306e92e2ca449c3969fae7ff2cda4f"}
Aug 01 00:08:57.051928 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:57.051879    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Aug 01 00:08:57.058273 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:57.058246    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 01 00:08:59.805275 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.805250    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:08:59.884722 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.884645    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/221cc730-d5c4-459b-87ed-877793f701fd-kube-rbac-proxy-sar-config\") pod \"221cc730-d5c4-459b-87ed-877793f701fd\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") "
Aug 01 00:08:59.884722 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.884693    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9kgh\" (UniqueName: \"kubernetes.io/projected/221cc730-d5c4-459b-87ed-877793f701fd-kube-api-access-t9kgh\") pod \"221cc730-d5c4-459b-87ed-877793f701fd\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") "
Aug 01 00:08:59.884722 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.884719    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221cc730-d5c4-459b-87ed-877793f701fd-proxy-tls\") pod \"221cc730-d5c4-459b-87ed-877793f701fd\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") "
Aug 01 00:08:59.884942 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.884759    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/221cc730-d5c4-459b-87ed-877793f701fd-cabundle-cert\") pod \"221cc730-d5c4-459b-87ed-877793f701fd\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") "
Aug 01 00:08:59.884942 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.884803    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/221cc730-d5c4-459b-87ed-877793f701fd-kserve-provision-location\") pod \"221cc730-d5c4-459b-87ed-877793f701fd\" (UID: \"221cc730-d5c4-459b-87ed-877793f701fd\") "
Aug 01 00:08:59.885167 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.885144    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221cc730-d5c4-459b-87ed-877793f701fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "221cc730-d5c4-459b-87ed-877793f701fd" (UID: "221cc730-d5c4-459b-87ed-877793f701fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:08:59.885240 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.885145    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221cc730-d5c4-459b-87ed-877793f701fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "221cc730-d5c4-459b-87ed-877793f701fd" (UID: "221cc730-d5c4-459b-87ed-877793f701fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:08:59.885240 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.885153    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221cc730-d5c4-459b-87ed-877793f701fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "221cc730-d5c4-459b-87ed-877793f701fd" (UID: "221cc730-d5c4-459b-87ed-877793f701fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:08:59.886887 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.886861    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/221cc730-d5c4-459b-87ed-877793f701fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "221cc730-d5c4-459b-87ed-877793f701fd" (UID: "221cc730-d5c4-459b-87ed-877793f701fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:08:59.887030 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.886919    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221cc730-d5c4-459b-87ed-877793f701fd-kube-api-access-t9kgh" (OuterVolumeSpecName: "kube-api-access-t9kgh") pod "221cc730-d5c4-459b-87ed-877793f701fd" (UID: "221cc730-d5c4-459b-87ed-877793f701fd"). InnerVolumeSpecName "kube-api-access-t9kgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:08:59.986034 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.985995    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/221cc730-d5c4-459b-87ed-877793f701fd-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:08:59.986034 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.986026    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/221cc730-d5c4-459b-87ed-877793f701fd-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:08:59.986034 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.986040    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/221cc730-d5c4-459b-87ed-877793f701fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:08:59.986264 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.986054    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9kgh\" (UniqueName: \"kubernetes.io/projected/221cc730-d5c4-459b-87ed-877793f701fd-kube-api-access-t9kgh\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:08:59.986264 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:08:59.986067    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221cc730-d5c4-459b-87ed-877793f701fd-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:09:00.368479 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.368446    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"]
Aug 01 00:09:00.368737 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.368725    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="kube-rbac-proxy"
Aug 01 00:09:00.368817 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.368738    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="kube-rbac-proxy"
Aug 01 00:09:00.368817 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.368758    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="storage-initializer"
Aug 01 00:09:00.368817 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.368764    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="storage-initializer"
Aug 01 00:09:00.368817 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.368771    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="kserve-container"
Aug 01 00:09:00.368817 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.368777    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="kserve-container"
Aug 01 00:09:00.369009 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.368826    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="kserve-container"
Aug 01 00:09:00.369009 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.368835    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="221cc730-d5c4-459b-87ed-877793f701fd" containerName="kube-rbac-proxy"
Aug 01 00:09:00.372005 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.371985    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:00.374502 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.374480    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Aug 01 00:09:00.374502 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.374497    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Aug 01 00:09:00.383522 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.383487    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"]
Aug 01 00:09:00.431535 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.431504    2584 generic.go:356] "Generic (PLEG): container finished" podID="221cc730-d5c4-459b-87ed-877793f701fd" containerID="511d75acc839bf33a6fb3ae086b859b30842d582b4fea01d1ab3107e69e1435d" exitCode=0
Aug 01 00:09:00.431722 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.431590    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"
Aug 01 00:09:00.431722 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.431595    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" event={"ID":"221cc730-d5c4-459b-87ed-877793f701fd","Type":"ContainerDied","Data":"511d75acc839bf33a6fb3ae086b859b30842d582b4fea01d1ab3107e69e1435d"}
Aug 01 00:09:00.431722 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.431681    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv" event={"ID":"221cc730-d5c4-459b-87ed-877793f701fd","Type":"ContainerDied","Data":"446bc572ada7dcb2ef00fde5ab4b16cb2b3ec5d7fd3b2f389979f735d9658b2a"}
Aug 01 00:09:00.431722 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.431720    2584 scope.go:117] "RemoveContainer" containerID="d22665c5bb422493eb0cfc770d19261a8c306e92e2ca449c3969fae7ff2cda4f"
Aug 01 00:09:00.440918 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.440901    2584 scope.go:117] "RemoveContainer" containerID="511d75acc839bf33a6fb3ae086b859b30842d582b4fea01d1ab3107e69e1435d"
Aug 01 00:09:00.448483 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.448463    2584 scope.go:117] "RemoveContainer" containerID="7909e4b5d137c4d16670c7459869abfd1117bfdb3560007bbfa95c0bf7b298f1"
Aug 01 00:09:00.455120 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.455011    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"]
Aug 01 00:09:00.456606 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.456583    2584 scope.go:117] "RemoveContainer" containerID="d22665c5bb422493eb0cfc770d19261a8c306e92e2ca449c3969fae7ff2cda4f"
Aug 01 00:09:00.456917 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:09:00.456899    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22665c5bb422493eb0cfc770d19261a8c306e92e2ca449c3969fae7ff2cda4f\": container with ID starting with d22665c5bb422493eb0cfc770d19261a8c306e92e2ca449c3969fae7ff2cda4f not found: ID does not exist" containerID="d22665c5bb422493eb0cfc770d19261a8c306e92e2ca449c3969fae7ff2cda4f"
Aug 01 00:09:00.456999 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.456926    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22665c5bb422493eb0cfc770d19261a8c306e92e2ca449c3969fae7ff2cda4f"} err="failed to get container status \"d22665c5bb422493eb0cfc770d19261a8c306e92e2ca449c3969fae7ff2cda4f\": rpc error: code = NotFound desc = could not find container \"d22665c5bb422493eb0cfc770d19261a8c306e92e2ca449c3969fae7ff2cda4f\": container with ID starting with d22665c5bb422493eb0cfc770d19261a8c306e92e2ca449c3969fae7ff2cda4f not found: ID does not exist"
Aug 01 00:09:00.456999 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.456948    2584 scope.go:117] "RemoveContainer" containerID="511d75acc839bf33a6fb3ae086b859b30842d582b4fea01d1ab3107e69e1435d"
Aug 01 00:09:00.457217 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:09:00.457201    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511d75acc839bf33a6fb3ae086b859b30842d582b4fea01d1ab3107e69e1435d\": container with ID starting with 511d75acc839bf33a6fb3ae086b859b30842d582b4fea01d1ab3107e69e1435d not found: ID does not exist" containerID="511d75acc839bf33a6fb3ae086b859b30842d582b4fea01d1ab3107e69e1435d"
Aug 01 00:09:00.457256 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.457222    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511d75acc839bf33a6fb3ae086b859b30842d582b4fea01d1ab3107e69e1435d"} err="failed to get container status \"511d75acc839bf33a6fb3ae086b859b30842d582b4fea01d1ab3107e69e1435d\": rpc error: code = NotFound desc = could not find container \"511d75acc839bf33a6fb3ae086b859b30842d582b4fea01d1ab3107e69e1435d\": container with ID starting with 511d75acc839bf33a6fb3ae086b859b30842d582b4fea01d1ab3107e69e1435d not found: ID does not exist"
Aug 01 00:09:00.457256 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.457234    2584 scope.go:117] "RemoveContainer" containerID="7909e4b5d137c4d16670c7459869abfd1117bfdb3560007bbfa95c0bf7b298f1"
Aug 01 00:09:00.457463 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:09:00.457445    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7909e4b5d137c4d16670c7459869abfd1117bfdb3560007bbfa95c0bf7b298f1\": container with ID starting with 7909e4b5d137c4d16670c7459869abfd1117bfdb3560007bbfa95c0bf7b298f1 not found: ID does not exist" containerID="7909e4b5d137c4d16670c7459869abfd1117bfdb3560007bbfa95c0bf7b298f1"
Aug 01 00:09:00.457500 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.457470    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7909e4b5d137c4d16670c7459869abfd1117bfdb3560007bbfa95c0bf7b298f1"} err="failed to get container status \"7909e4b5d137c4d16670c7459869abfd1117bfdb3560007bbfa95c0bf7b298f1\": rpc error: code = NotFound desc = could not find container \"7909e4b5d137c4d16670c7459869abfd1117bfdb3560007bbfa95c0bf7b298f1\": container with ID starting with 7909e4b5d137c4d16670c7459869abfd1117bfdb3560007bbfa95c0bf7b298f1 not found: ID does not exist"
Aug 01 00:09:00.461032 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.461010    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-64flv"]
Aug 01 00:09:00.490028 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.490004    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:00.490141 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.490041    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqngz\" (UniqueName: \"kubernetes.io/projected/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kube-api-access-sqngz\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:00.490141 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.490102    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:00.490141 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.490122    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:00.490259 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.490149    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:00.591302 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.591265    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:00.591302 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.591306    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:00.591496 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.591332    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:00.591496 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.591358    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:00.591496 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.591392    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqngz\" (UniqueName: \"kubernetes.io/projected/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kube-api-access-sqngz\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:00.591496 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:09:00.591429    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Aug 01 00:09:00.591660 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:09:00.591517    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-proxy-tls podName:a05f435f-b23b-4a2f-a0fe-a91d4f2920d5 nodeName:}" failed. No retries permitted until 2026-08-01 00:09:01.091496443 +0000 UTC m=+1375.965140031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" (UID: "a05f435f-b23b-4a2f-a0fe-a91d4f2920d5") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Aug 01 00:09:00.591816 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.591797    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:00.591947 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.591930    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:00.592026 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.592008    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:00.599712 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:00.599686    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqngz\" (UniqueName: \"kubernetes.io/projected/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kube-api-access-sqngz\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:01.095957 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:01.095915    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:01.098320 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:01.098300    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:01.290344 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:01.290306    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:01.420907 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:01.420879    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"]
Aug 01 00:09:01.422917 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:09:01.422892    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05f435f_b23b_4a2f_a0fe_a91d4f2920d5.slice/crio-672b37273224c862e6648659cb8be561d83639952f71619a10db783c8b5418a5 WatchSource:0}: Error finding container 672b37273224c862e6648659cb8be561d83639952f71619a10db783c8b5418a5: Status 404 returned error can't find the container with id 672b37273224c862e6648659cb8be561d83639952f71619a10db783c8b5418a5
Aug 01 00:09:01.439293 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:01.439265    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" event={"ID":"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5","Type":"ContainerStarted","Data":"672b37273224c862e6648659cb8be561d83639952f71619a10db783c8b5418a5"}
Aug 01 00:09:01.571635 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:01.571601    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="221cc730-d5c4-459b-87ed-877793f701fd" path="/var/lib/kubelet/pods/221cc730-d5c4-459b-87ed-877793f701fd/volumes"
Aug 01 00:09:02.444160 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:02.444127    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" event={"ID":"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5","Type":"ContainerStarted","Data":"f2a107767e7cab38c4e8cb5eb0c4ca0f466b2f428ec8897831c27537854c1abc"}
Aug 01 00:09:05.457158 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:05.457077    2584 generic.go:356] "Generic (PLEG): container finished" podID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerID="f2a107767e7cab38c4e8cb5eb0c4ca0f466b2f428ec8897831c27537854c1abc" exitCode=0
Aug 01 00:09:05.457493 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:05.457153    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" event={"ID":"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5","Type":"ContainerDied","Data":"f2a107767e7cab38c4e8cb5eb0c4ca0f466b2f428ec8897831c27537854c1abc"}
Aug 01 00:09:06.463152 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:06.463112    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" event={"ID":"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5","Type":"ContainerStarted","Data":"241d6d3ff74cd8b4685adf1c65eb5b5f16ab91a5328debe748bf46055ee264ac"}
Aug 01 00:09:06.463593 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:06.463175    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" event={"ID":"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5","Type":"ContainerStarted","Data":"4fd65236f1c383f6942404fdd015340513dd7d5d78cb9347780a2e55cca94a67"}
Aug 01 00:09:06.463593 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:06.463497    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:06.463593 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:06.463543    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:06.481481 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:06.481433    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" podStartSLOduration=2.790964781 podStartE2EDuration="6.481419159s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.690454378s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:09:06.480685432 +0000 UTC m=+1381.354329057" watchObservedRunningTime="2026-08-01 00:09:06.481419159 +0000 UTC m=+1381.355062769"
Aug 01 00:09:12.472666 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:12.472633    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:09:42.473910 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:42.473868    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 01 00:09:52.473827 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:09:52.473786    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 01 00:10:02.474147 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:02.474053    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 01 00:10:12.473798 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:12.473755    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 01 00:10:22.521442 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:22.521410    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:10:30.440770 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:30.440736    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"]
Aug 01 00:10:30.441301 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:30.441013    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="kserve-container" containerID="cri-o://4fd65236f1c383f6942404fdd015340513dd7d5d78cb9347780a2e55cca94a67" gracePeriod=30
Aug 01 00:10:30.441301 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:30.441055    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="kube-rbac-proxy" containerID="cri-o://241d6d3ff74cd8b4685adf1c65eb5b5f16ab91a5328debe748bf46055ee264ac" gracePeriod=30
Aug 01 00:10:30.833464 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:30.833430    2584 generic.go:356] "Generic (PLEG): container finished" podID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerID="241d6d3ff74cd8b4685adf1c65eb5b5f16ab91a5328debe748bf46055ee264ac" exitCode=2
Aug 01 00:10:30.833631 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:30.833502    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" event={"ID":"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5","Type":"ContainerDied","Data":"241d6d3ff74cd8b4685adf1c65eb5b5f16ab91a5328debe748bf46055ee264ac"}
Aug 01 00:10:32.467851 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:32.467805    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Aug 01 00:10:32.473442 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:32.473413    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 01 00:10:35.588680 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.588651    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:10:35.763922 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.763851    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kube-rbac-proxy-sar-config\") pod \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") "
Aug 01 00:10:35.763922 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.763886    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-cabundle-cert\") pod \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") "
Aug 01 00:10:35.763922 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.763919    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqngz\" (UniqueName: \"kubernetes.io/projected/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kube-api-access-sqngz\") pod \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") "
Aug 01 00:10:35.764215 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.763947    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kserve-provision-location\") pod \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") "
Aug 01 00:10:35.764215 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.764007    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-proxy-tls\") pod \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\" (UID: \"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5\") "
Aug 01 00:10:35.764333 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.764287    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" (UID: "a05f435f-b23b-4a2f-a0fe-a91d4f2920d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:10:35.764333 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.764297    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" (UID: "a05f435f-b23b-4a2f-a0fe-a91d4f2920d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:10:35.764403 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.764342    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" (UID: "a05f435f-b23b-4a2f-a0fe-a91d4f2920d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:10:35.765886 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.765864    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kube-api-access-sqngz" (OuterVolumeSpecName: "kube-api-access-sqngz") pod "a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" (UID: "a05f435f-b23b-4a2f-a0fe-a91d4f2920d5"). InnerVolumeSpecName "kube-api-access-sqngz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:10:35.765955 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.765865    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" (UID: "a05f435f-b23b-4a2f-a0fe-a91d4f2920d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:10:35.854867 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.854832    2584 generic.go:356] "Generic (PLEG): container finished" podID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerID="4fd65236f1c383f6942404fdd015340513dd7d5d78cb9347780a2e55cca94a67" exitCode=0
Aug 01 00:10:35.855064 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.854889    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" event={"ID":"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5","Type":"ContainerDied","Data":"4fd65236f1c383f6942404fdd015340513dd7d5d78cb9347780a2e55cca94a67"}
Aug 01 00:10:35.855064 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.854896    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"
Aug 01 00:10:35.855064 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.854941    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk" event={"ID":"a05f435f-b23b-4a2f-a0fe-a91d4f2920d5","Type":"ContainerDied","Data":"672b37273224c862e6648659cb8be561d83639952f71619a10db783c8b5418a5"}
Aug 01 00:10:35.855064 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.854994    2584 scope.go:117] "RemoveContainer" containerID="241d6d3ff74cd8b4685adf1c65eb5b5f16ab91a5328debe748bf46055ee264ac"
Aug 01 00:10:35.866004 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.864632    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:10:35.866004 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.864660    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:10:35.866004 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.864675    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqngz\" (UniqueName: \"kubernetes.io/projected/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kube-api-access-sqngz\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:10:35.866004 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.864689    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:10:35.866004 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.864703    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:10:35.867436 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.867418    2584 scope.go:117] "RemoveContainer" containerID="4fd65236f1c383f6942404fdd015340513dd7d5d78cb9347780a2e55cca94a67"
Aug 01 00:10:35.875356 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.875335    2584 scope.go:117] "RemoveContainer" containerID="f2a107767e7cab38c4e8cb5eb0c4ca0f466b2f428ec8897831c27537854c1abc"
Aug 01 00:10:35.877067 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.877047    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"]
Aug 01 00:10:35.881217 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.881199    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-szxkk"]
Aug 01 00:10:35.883348 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.883329    2584 scope.go:117] "RemoveContainer" containerID="241d6d3ff74cd8b4685adf1c65eb5b5f16ab91a5328debe748bf46055ee264ac"
Aug 01 00:10:35.883602 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:10:35.883584    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"241d6d3ff74cd8b4685adf1c65eb5b5f16ab91a5328debe748bf46055ee264ac\": container with ID starting with 241d6d3ff74cd8b4685adf1c65eb5b5f16ab91a5328debe748bf46055ee264ac not found: ID does not exist" containerID="241d6d3ff74cd8b4685adf1c65eb5b5f16ab91a5328debe748bf46055ee264ac"
Aug 01 00:10:35.883650 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.883612    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241d6d3ff74cd8b4685adf1c65eb5b5f16ab91a5328debe748bf46055ee264ac"} err="failed to get container status \"241d6d3ff74cd8b4685adf1c65eb5b5f16ab91a5328debe748bf46055ee264ac\": rpc error: code = NotFound desc = could not find container \"241d6d3ff74cd8b4685adf1c65eb5b5f16ab91a5328debe748bf46055ee264ac\": container with ID starting with 241d6d3ff74cd8b4685adf1c65eb5b5f16ab91a5328debe748bf46055ee264ac not found: ID does not exist"
Aug 01 00:10:35.883650 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.883630    2584 scope.go:117] "RemoveContainer" containerID="4fd65236f1c383f6942404fdd015340513dd7d5d78cb9347780a2e55cca94a67"
Aug 01 00:10:35.883874 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:10:35.883856    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd65236f1c383f6942404fdd015340513dd7d5d78cb9347780a2e55cca94a67\": container with ID starting with 4fd65236f1c383f6942404fdd015340513dd7d5d78cb9347780a2e55cca94a67 not found: ID does not exist" containerID="4fd65236f1c383f6942404fdd015340513dd7d5d78cb9347780a2e55cca94a67"
Aug 01 00:10:35.883936 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.883884    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd65236f1c383f6942404fdd015340513dd7d5d78cb9347780a2e55cca94a67"} err="failed to get container status \"4fd65236f1c383f6942404fdd015340513dd7d5d78cb9347780a2e55cca94a67\": rpc error: code = NotFound desc = could not find container \"4fd65236f1c383f6942404fdd015340513dd7d5d78cb9347780a2e55cca94a67\": container with ID starting with 4fd65236f1c383f6942404fdd015340513dd7d5d78cb9347780a2e55cca94a67 not found: ID does not exist"
Aug 01 00:10:35.883936 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.883908    2584 scope.go:117] "RemoveContainer" containerID="f2a107767e7cab38c4e8cb5eb0c4ca0f466b2f428ec8897831c27537854c1abc"
Aug 01 00:10:35.884185 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:10:35.884170    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a107767e7cab38c4e8cb5eb0c4ca0f466b2f428ec8897831c27537854c1abc\": container with ID starting with f2a107767e7cab38c4e8cb5eb0c4ca0f466b2f428ec8897831c27537854c1abc not found: ID does not exist" containerID="f2a107767e7cab38c4e8cb5eb0c4ca0f466b2f428ec8897831c27537854c1abc"
Aug 01 00:10:35.884225 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:35.884191    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a107767e7cab38c4e8cb5eb0c4ca0f466b2f428ec8897831c27537854c1abc"} err="failed to get container status \"f2a107767e7cab38c4e8cb5eb0c4ca0f466b2f428ec8897831c27537854c1abc\": rpc error: code = NotFound desc = could not find container \"f2a107767e7cab38c4e8cb5eb0c4ca0f466b2f428ec8897831c27537854c1abc\": container with ID starting with f2a107767e7cab38c4e8cb5eb0c4ca0f466b2f428ec8897831c27537854c1abc not found: ID does not exist"
Aug 01 00:10:37.571390 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:37.571358    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" path="/var/lib/kubelet/pods/a05f435f-b23b-4a2f-a0fe-a91d4f2920d5/volumes"
Aug 01 00:10:55.490394 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.490360    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"]
Aug 01 00:10:55.490792 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.490750    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="storage-initializer"
Aug 01 00:10:55.490792 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.490762    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="storage-initializer"
Aug 01 00:10:55.490792 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.490777    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="kserve-container"
Aug 01 00:10:55.490792 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.490783    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="kserve-container"
Aug 01 00:10:55.490921 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.490801    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="kube-rbac-proxy"
Aug 01 00:10:55.490921 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.490807    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="kube-rbac-proxy"
Aug 01 00:10:55.490921 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.490856    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="kube-rbac-proxy"
Aug 01 00:10:55.490921 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.490862    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a05f435f-b23b-4a2f-a0fe-a91d4f2920d5" containerName="kserve-container"
Aug 01 00:10:55.495404 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.495385    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.497985 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.497946    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Aug 01 00:10:55.497985 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.497951    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:10:55.498175 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.498046    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:10:55.498175 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.498113    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:10:55.498481 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.498457    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Aug 01 00:10:55.499077 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.499054    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:10:55.499162 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.499062    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:10:55.505137 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.505113    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"]
Aug 01 00:10:55.618015 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.617982    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d71eebe4-ba10-4b97-aa90-78ab77659892-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.618182 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.618039    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d71eebe4-ba10-4b97-aa90-78ab77659892-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.618182 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.618088    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qlm2\" (UniqueName: \"kubernetes.io/projected/d71eebe4-ba10-4b97-aa90-78ab77659892-kube-api-access-9qlm2\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.618182 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.618148    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d71eebe4-ba10-4b97-aa90-78ab77659892-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.618334 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.618186    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d71eebe4-ba10-4b97-aa90-78ab77659892-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.719075 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.719044    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d71eebe4-ba10-4b97-aa90-78ab77659892-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.719230 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.719097    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d71eebe4-ba10-4b97-aa90-78ab77659892-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.719230 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.719125    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d71eebe4-ba10-4b97-aa90-78ab77659892-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.719230 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.719161    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qlm2\" (UniqueName: \"kubernetes.io/projected/d71eebe4-ba10-4b97-aa90-78ab77659892-kube-api-access-9qlm2\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.719230 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.719188    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d71eebe4-ba10-4b97-aa90-78ab77659892-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.719598 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.719579    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d71eebe4-ba10-4b97-aa90-78ab77659892-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.719771 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.719750    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d71eebe4-ba10-4b97-aa90-78ab77659892-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.719840 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.719799    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d71eebe4-ba10-4b97-aa90-78ab77659892-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.721698 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.721678    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d71eebe4-ba10-4b97-aa90-78ab77659892-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.727827 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.727803    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qlm2\" (UniqueName: \"kubernetes.io/projected/d71eebe4-ba10-4b97-aa90-78ab77659892-kube-api-access-9qlm2\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.807585 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.807552    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:10:55.938611 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:55.938588    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"]
Aug 01 00:10:55.940308 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:10:55.940285    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71eebe4_ba10_4b97_aa90_78ab77659892.slice/crio-8c3da7e472119ed188a0d95b26b970e2a9235ea634a59cc0a8342135ff07b3c8 WatchSource:0}: Error finding container 8c3da7e472119ed188a0d95b26b970e2a9235ea634a59cc0a8342135ff07b3c8: Status 404 returned error can't find the container with id 8c3da7e472119ed188a0d95b26b970e2a9235ea634a59cc0a8342135ff07b3c8
Aug 01 00:10:56.946436 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:56.946397    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc" event={"ID":"d71eebe4-ba10-4b97-aa90-78ab77659892","Type":"ContainerStarted","Data":"0d28fd249a7c8351e4478195a5f6df77856bbba81580710765f2819027c420ac"}
Aug 01 00:10:56.946813 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:10:56.946491    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc" event={"ID":"d71eebe4-ba10-4b97-aa90-78ab77659892","Type":"ContainerStarted","Data":"8c3da7e472119ed188a0d95b26b970e2a9235ea634a59cc0a8342135ff07b3c8"}
Aug 01 00:11:05.605757 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:11:05.605730    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:11:05.608165 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:11:05.608138    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:11:25.065721 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:11:25.065682    2584 generic.go:356] "Generic (PLEG): container finished" podID="d71eebe4-ba10-4b97-aa90-78ab77659892" containerID="0d28fd249a7c8351e4478195a5f6df77856bbba81580710765f2819027c420ac" exitCode=0
Aug 01 00:11:25.066262 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:11:25.065761    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc" event={"ID":"d71eebe4-ba10-4b97-aa90-78ab77659892","Type":"ContainerDied","Data":"0d28fd249a7c8351e4478195a5f6df77856bbba81580710765f2819027c420ac"}
Aug 01 00:11:26.072882 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:11:26.072847    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc" event={"ID":"d71eebe4-ba10-4b97-aa90-78ab77659892","Type":"ContainerStarted","Data":"0af76a0a815fafcdb05b5c06a8f8b4902fe304f72c8decda07c1e8ec4a686097"}
Aug 01 00:11:26.073325 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:11:26.072907    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc" event={"ID":"d71eebe4-ba10-4b97-aa90-78ab77659892","Type":"ContainerStarted","Data":"0e5f24cc256733415c0c5a50485372be40d05f672d3d110f25bb2c01ac1180b5"}
Aug 01 00:11:26.073325 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:11:26.073114    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:11:26.092229 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:11:26.092173    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc" podStartSLOduration=2.582981611 podStartE2EDuration="31.092156196s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.509174585s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:11:26.090247728 +0000 UTC m=+1520.963891339" watchObservedRunningTime="2026-08-01 00:11:26.092156196 +0000 UTC m=+1520.965799806"
Aug 01 00:11:27.078417 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:11:27.078383    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:11:33.087565 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:11:33.087535    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:12:58.568836 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:12:58.568763    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:13:09.607139 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:09.607096    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc" podUID="d71eebe4-ba10-4b97-aa90-78ab77659892" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Aug 01 00:13:11.683113 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:11.683077    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"]
Aug 01 00:13:11.683480 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:11.683312    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc" podUID="d71eebe4-ba10-4b97-aa90-78ab77659892" containerName="kserve-container" containerID="cri-o://0e5f24cc256733415c0c5a50485372be40d05f672d3d110f25bb2c01ac1180b5" gracePeriod=30
Aug 01 00:13:11.683480 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:11.683347    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc" podUID="d71eebe4-ba10-4b97-aa90-78ab77659892" containerName="kube-rbac-proxy" containerID="cri-o://0af76a0a815fafcdb05b5c06a8f8b4902fe304f72c8decda07c1e8ec4a686097" gracePeriod=30
Aug 01 00:13:12.526386 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:12.526345    2584 generic.go:356] "Generic (PLEG): container finished" podID="d71eebe4-ba10-4b97-aa90-78ab77659892" containerID="0af76a0a815fafcdb05b5c06a8f8b4902fe304f72c8decda07c1e8ec4a686097" exitCode=2
Aug 01 00:13:12.526554 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:12.526420    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc" event={"ID":"d71eebe4-ba10-4b97-aa90-78ab77659892","Type":"ContainerDied","Data":"0af76a0a815fafcdb05b5c06a8f8b4902fe304f72c8decda07c1e8ec4a686097"}
Aug 01 00:13:13.025613 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.025590    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:13:13.136419 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.136339    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d71eebe4-ba10-4b97-aa90-78ab77659892-kserve-provision-location\") pod \"d71eebe4-ba10-4b97-aa90-78ab77659892\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") "
Aug 01 00:13:13.136419 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.136393    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d71eebe4-ba10-4b97-aa90-78ab77659892-kube-rbac-proxy-sar-config\") pod \"d71eebe4-ba10-4b97-aa90-78ab77659892\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") "
Aug 01 00:13:13.136592 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.136437    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d71eebe4-ba10-4b97-aa90-78ab77659892-cabundle-cert\") pod \"d71eebe4-ba10-4b97-aa90-78ab77659892\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") "
Aug 01 00:13:13.136592 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.136514    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d71eebe4-ba10-4b97-aa90-78ab77659892-proxy-tls\") pod \"d71eebe4-ba10-4b97-aa90-78ab77659892\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") "
Aug 01 00:13:13.136592 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.136540    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qlm2\" (UniqueName: \"kubernetes.io/projected/d71eebe4-ba10-4b97-aa90-78ab77659892-kube-api-access-9qlm2\") pod \"d71eebe4-ba10-4b97-aa90-78ab77659892\" (UID: \"d71eebe4-ba10-4b97-aa90-78ab77659892\") "
Aug 01 00:13:13.136802 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.136772    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71eebe4-ba10-4b97-aa90-78ab77659892-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d71eebe4-ba10-4b97-aa90-78ab77659892" (UID: "d71eebe4-ba10-4b97-aa90-78ab77659892"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:13:13.136941 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.136917    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71eebe4-ba10-4b97-aa90-78ab77659892-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d71eebe4-ba10-4b97-aa90-78ab77659892" (UID: "d71eebe4-ba10-4b97-aa90-78ab77659892"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:13:13.138643 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.138618    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d71eebe4-ba10-4b97-aa90-78ab77659892-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d71eebe4-ba10-4b97-aa90-78ab77659892" (UID: "d71eebe4-ba10-4b97-aa90-78ab77659892"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:13:13.138764 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.138745    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71eebe4-ba10-4b97-aa90-78ab77659892-kube-api-access-9qlm2" (OuterVolumeSpecName: "kube-api-access-9qlm2") pod "d71eebe4-ba10-4b97-aa90-78ab77659892" (UID: "d71eebe4-ba10-4b97-aa90-78ab77659892"). InnerVolumeSpecName "kube-api-access-9qlm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:13:13.141119 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.141100    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71eebe4-ba10-4b97-aa90-78ab77659892-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d71eebe4-ba10-4b97-aa90-78ab77659892" (UID: "d71eebe4-ba10-4b97-aa90-78ab77659892"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:13.237285 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.237241    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d71eebe4-ba10-4b97-aa90-78ab77659892-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:13:13.237285 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.237282    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d71eebe4-ba10-4b97-aa90-78ab77659892-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:13:13.237285 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.237297    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d71eebe4-ba10-4b97-aa90-78ab77659892-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:13:13.237484 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.237305    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qlm2\" (UniqueName: \"kubernetes.io/projected/d71eebe4-ba10-4b97-aa90-78ab77659892-kube-api-access-9qlm2\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:13:13.237484 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.237314    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d71eebe4-ba10-4b97-aa90-78ab77659892-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:13:13.532515 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.532432    2584 generic.go:356] "Generic (PLEG): container finished" podID="d71eebe4-ba10-4b97-aa90-78ab77659892" containerID="0e5f24cc256733415c0c5a50485372be40d05f672d3d110f25bb2c01ac1180b5" exitCode=0
Aug 01 00:13:13.532515 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.532494    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"
Aug 01 00:13:13.532515 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.532502    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc" event={"ID":"d71eebe4-ba10-4b97-aa90-78ab77659892","Type":"ContainerDied","Data":"0e5f24cc256733415c0c5a50485372be40d05f672d3d110f25bb2c01ac1180b5"}
Aug 01 00:13:13.532780 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.532549    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc" event={"ID":"d71eebe4-ba10-4b97-aa90-78ab77659892","Type":"ContainerDied","Data":"8c3da7e472119ed188a0d95b26b970e2a9235ea634a59cc0a8342135ff07b3c8"}
Aug 01 00:13:13.532780 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.532577    2584 scope.go:117] "RemoveContainer" containerID="0af76a0a815fafcdb05b5c06a8f8b4902fe304f72c8decda07c1e8ec4a686097"
Aug 01 00:13:13.541494 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.541477    2584 scope.go:117] "RemoveContainer" containerID="0e5f24cc256733415c0c5a50485372be40d05f672d3d110f25bb2c01ac1180b5"
Aug 01 00:13:13.548612 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.548597    2584 scope.go:117] "RemoveContainer" containerID="0d28fd249a7c8351e4478195a5f6df77856bbba81580710765f2819027c420ac"
Aug 01 00:13:13.555781 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.555762    2584 scope.go:117] "RemoveContainer" containerID="0af76a0a815fafcdb05b5c06a8f8b4902fe304f72c8decda07c1e8ec4a686097"
Aug 01 00:13:13.556018 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:13:13.556001    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af76a0a815fafcdb05b5c06a8f8b4902fe304f72c8decda07c1e8ec4a686097\": container with ID starting with 0af76a0a815fafcdb05b5c06a8f8b4902fe304f72c8decda07c1e8ec4a686097 not found: ID does not exist" containerID="0af76a0a815fafcdb05b5c06a8f8b4902fe304f72c8decda07c1e8ec4a686097"
Aug 01 00:13:13.556056 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.556029    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af76a0a815fafcdb05b5c06a8f8b4902fe304f72c8decda07c1e8ec4a686097"} err="failed to get container status \"0af76a0a815fafcdb05b5c06a8f8b4902fe304f72c8decda07c1e8ec4a686097\": rpc error: code = NotFound desc = could not find container \"0af76a0a815fafcdb05b5c06a8f8b4902fe304f72c8decda07c1e8ec4a686097\": container with ID starting with 0af76a0a815fafcdb05b5c06a8f8b4902fe304f72c8decda07c1e8ec4a686097 not found: ID does not exist"
Aug 01 00:13:13.556056 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.556045    2584 scope.go:117] "RemoveContainer" containerID="0e5f24cc256733415c0c5a50485372be40d05f672d3d110f25bb2c01ac1180b5"
Aug 01 00:13:13.556266 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:13:13.556248    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5f24cc256733415c0c5a50485372be40d05f672d3d110f25bb2c01ac1180b5\": container with ID starting with 0e5f24cc256733415c0c5a50485372be40d05f672d3d110f25bb2c01ac1180b5 not found: ID does not exist" containerID="0e5f24cc256733415c0c5a50485372be40d05f672d3d110f25bb2c01ac1180b5"
Aug 01 00:13:13.556310 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.556272    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5f24cc256733415c0c5a50485372be40d05f672d3d110f25bb2c01ac1180b5"} err="failed to get container status \"0e5f24cc256733415c0c5a50485372be40d05f672d3d110f25bb2c01ac1180b5\": rpc error: code = NotFound desc = could not find container \"0e5f24cc256733415c0c5a50485372be40d05f672d3d110f25bb2c01ac1180b5\": container with ID starting with 0e5f24cc256733415c0c5a50485372be40d05f672d3d110f25bb2c01ac1180b5 not found: ID does not exist"
Aug 01 00:13:13.556310 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.556291    2584 scope.go:117] "RemoveContainer" containerID="0d28fd249a7c8351e4478195a5f6df77856bbba81580710765f2819027c420ac"
Aug 01 00:13:13.556504 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:13:13.556489    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d28fd249a7c8351e4478195a5f6df77856bbba81580710765f2819027c420ac\": container with ID starting with 0d28fd249a7c8351e4478195a5f6df77856bbba81580710765f2819027c420ac not found: ID does not exist" containerID="0d28fd249a7c8351e4478195a5f6df77856bbba81580710765f2819027c420ac"
Aug 01 00:13:13.556541 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.556509    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d28fd249a7c8351e4478195a5f6df77856bbba81580710765f2819027c420ac"} err="failed to get container status \"0d28fd249a7c8351e4478195a5f6df77856bbba81580710765f2819027c420ac\": rpc error: code = NotFound desc = could not find container \"0d28fd249a7c8351e4478195a5f6df77856bbba81580710765f2819027c420ac\": container with ID starting with 0d28fd249a7c8351e4478195a5f6df77856bbba81580710765f2819027c420ac not found: ID does not exist"
Aug 01 00:13:13.570317 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.570298    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"]
Aug 01 00:13:13.575700 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:13.575677    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fwdfc"]
Aug 01 00:13:15.570847 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:13:15.570816    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d71eebe4-ba10-4b97-aa90-78ab77659892" path="/var/lib/kubelet/pods/d71eebe4-ba10-4b97-aa90-78ab77659892/volumes"
Aug 01 00:14:05.783361 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.783326    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"]
Aug 01 00:14:05.783781 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.783650    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d71eebe4-ba10-4b97-aa90-78ab77659892" containerName="storage-initializer"
Aug 01 00:14:05.783781 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.783667    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d71eebe4-ba10-4b97-aa90-78ab77659892" containerName="storage-initializer"
Aug 01 00:14:05.783781 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.783713    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d71eebe4-ba10-4b97-aa90-78ab77659892" containerName="kserve-container"
Aug 01 00:14:05.783781 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.783723    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d71eebe4-ba10-4b97-aa90-78ab77659892" containerName="kserve-container"
Aug 01 00:14:05.783781 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.783736    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d71eebe4-ba10-4b97-aa90-78ab77659892" containerName="kube-rbac-proxy"
Aug 01 00:14:05.783781 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.783744    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d71eebe4-ba10-4b97-aa90-78ab77659892" containerName="kube-rbac-proxy"
Aug 01 00:14:05.784010 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.783833    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d71eebe4-ba10-4b97-aa90-78ab77659892" containerName="kube-rbac-proxy"
Aug 01 00:14:05.784010 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.783849    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d71eebe4-ba10-4b97-aa90-78ab77659892" containerName="kserve-container"
Aug 01 00:14:05.788290 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.788269    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:05.791043 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.791024    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:14:05.791178 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.791150    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:14:05.791235 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.791195    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:14:05.791436 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.791423    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:14:05.792084 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.792068    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Aug 01 00:14:05.792162 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.792073    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Aug 01 00:14:05.792229 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.792075    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:14:05.796257 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.796235    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"]
Aug 01 00:14:05.875290 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.875259    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/089e861c-06db-4d39-bee9-de551706c4c1-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:05.875457 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.875302    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/089e861c-06db-4d39-bee9-de551706c4c1-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:05.875457 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.875360    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb9kn\" (UniqueName: \"kubernetes.io/projected/089e861c-06db-4d39-bee9-de551706c4c1-kube-api-access-hb9kn\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:05.875457 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.875392    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/089e861c-06db-4d39-bee9-de551706c4c1-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:05.875457 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.875422    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/089e861c-06db-4d39-bee9-de551706c4c1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:05.975849 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.975816    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/089e861c-06db-4d39-bee9-de551706c4c1-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:05.976037 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.975857    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/089e861c-06db-4d39-bee9-de551706c4c1-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:05.976037 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.975880    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9kn\" (UniqueName: \"kubernetes.io/projected/089e861c-06db-4d39-bee9-de551706c4c1-kube-api-access-hb9kn\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:05.976037 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.975903    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/089e861c-06db-4d39-bee9-de551706c4c1-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:05.976037 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.975939    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/089e861c-06db-4d39-bee9-de551706c4c1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:05.976037 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:14:05.975989    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Aug 01 00:14:05.976286 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:14:05.976067    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089e861c-06db-4d39-bee9-de551706c4c1-proxy-tls podName:089e861c-06db-4d39-bee9-de551706c4c1 nodeName:}" failed. No retries permitted until 2026-08-01 00:14:06.476047069 +0000 UTC m=+1681.349690657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/089e861c-06db-4d39-bee9-de551706c4c1-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks" (UID: "089e861c-06db-4d39-bee9-de551706c4c1") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Aug 01 00:14:05.976379 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.976363    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/089e861c-06db-4d39-bee9-de551706c4c1-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:05.976555 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.976538    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/089e861c-06db-4d39-bee9-de551706c4c1-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:05.976592 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.976563    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/089e861c-06db-4d39-bee9-de551706c4c1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:05.986829 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:05.986807    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb9kn\" (UniqueName: \"kubernetes.io/projected/089e861c-06db-4d39-bee9-de551706c4c1-kube-api-access-hb9kn\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:06.480440 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:06.480388    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/089e861c-06db-4d39-bee9-de551706c4c1-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:06.482858 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:06.482833    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/089e861c-06db-4d39-bee9-de551706c4c1-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:06.699543 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:06.699507    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:06.827926 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:06.827897    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"]
Aug 01 00:14:06.831673 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:14:06.831592    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089e861c_06db_4d39_bee9_de551706c4c1.slice/crio-90404394da200bf1b6287b548aefaa3bb9e63127d8b22926e69895fd04b9494d WatchSource:0}: Error finding container 90404394da200bf1b6287b548aefaa3bb9e63127d8b22926e69895fd04b9494d: Status 404 returned error can't find the container with id 90404394da200bf1b6287b548aefaa3bb9e63127d8b22926e69895fd04b9494d
Aug 01 00:14:07.753425 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:07.753392    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks" event={"ID":"089e861c-06db-4d39-bee9-de551706c4c1","Type":"ContainerStarted","Data":"2afe9c3aa61f536d8d894b688f2eb4378882947e2fdff31760321ea698bbd449"}
Aug 01 00:14:07.753591 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:07.753447    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks" event={"ID":"089e861c-06db-4d39-bee9-de551706c4c1","Type":"ContainerStarted","Data":"90404394da200bf1b6287b548aefaa3bb9e63127d8b22926e69895fd04b9494d"}
Aug 01 00:14:15.787331 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:15.787296    2584 generic.go:356] "Generic (PLEG): container finished" podID="089e861c-06db-4d39-bee9-de551706c4c1" containerID="2afe9c3aa61f536d8d894b688f2eb4378882947e2fdff31760321ea698bbd449" exitCode=0
Aug 01 00:14:15.787741 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:15.787370    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks" event={"ID":"089e861c-06db-4d39-bee9-de551706c4c1","Type":"ContainerDied","Data":"2afe9c3aa61f536d8d894b688f2eb4378882947e2fdff31760321ea698bbd449"}
Aug 01 00:14:16.793833 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:16.793796    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks" event={"ID":"089e861c-06db-4d39-bee9-de551706c4c1","Type":"ContainerStarted","Data":"7619ee7aa29033026d1b48a3dc82f9e1bf1ce6dc6332c1f4333ac79c62eae7bd"}
Aug 01 00:14:16.794218 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:16.793879    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks" event={"ID":"089e861c-06db-4d39-bee9-de551706c4c1","Type":"ContainerStarted","Data":"290993594dd5d81f3ff99903b9521bc03022ae4420ace0d5cb3a0fff077fbe2c"}
Aug 01 00:14:16.794218 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:16.794125    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:16.813551 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:16.813503    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks" podStartSLOduration=3.008844907 podStartE2EDuration="11.813488867s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.80464396s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:14:16.811679324 +0000 UTC m=+1691.685322936" watchObservedRunningTime="2026-08-01 00:14:16.813488867 +0000 UTC m=+1691.687132476"
Aug 01 00:14:17.798037 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:17.798002    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:17.799565 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:17.799537    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks" podUID="089e861c-06db-4d39-bee9-de551706c4c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 01 00:14:18.803453 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:18.803413    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks" podUID="089e861c-06db-4d39-bee9-de551706c4c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 01 00:14:23.808226 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:23.808187    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:23.808873 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:23.808843    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks" podUID="089e861c-06db-4d39-bee9-de551706c4c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 01 00:14:33.809707 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:33.809677    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:40.985626 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:40.985595    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"]
Aug 01 00:14:40.986064 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:40.985876    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks" podUID="089e861c-06db-4d39-bee9-de551706c4c1" containerName="kserve-container" containerID="cri-o://290993594dd5d81f3ff99903b9521bc03022ae4420ace0d5cb3a0fff077fbe2c" gracePeriod=30
Aug 01 00:14:40.986064 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:40.985991    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks" podUID="089e861c-06db-4d39-bee9-de551706c4c1" containerName="kube-rbac-proxy" containerID="cri-o://7619ee7aa29033026d1b48a3dc82f9e1bf1ce6dc6332c1f4333ac79c62eae7bd" gracePeriod=30
Aug 01 00:14:41.905472 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:41.905431    2584 generic.go:356] "Generic (PLEG): container finished" podID="089e861c-06db-4d39-bee9-de551706c4c1" containerID="7619ee7aa29033026d1b48a3dc82f9e1bf1ce6dc6332c1f4333ac79c62eae7bd" exitCode=2
Aug 01 00:14:41.905654 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:41.905509    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks" event={"ID":"089e861c-06db-4d39-bee9-de551706c4c1","Type":"ContainerDied","Data":"7619ee7aa29033026d1b48a3dc82f9e1bf1ce6dc6332c1f4333ac79c62eae7bd"}
Aug 01 00:14:42.323759 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.323737    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:42.377654 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.377625    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb9kn\" (UniqueName: \"kubernetes.io/projected/089e861c-06db-4d39-bee9-de551706c4c1-kube-api-access-hb9kn\") pod \"089e861c-06db-4d39-bee9-de551706c4c1\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") "
Aug 01 00:14:42.377654 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.377659    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/089e861c-06db-4d39-bee9-de551706c4c1-proxy-tls\") pod \"089e861c-06db-4d39-bee9-de551706c4c1\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") "
Aug 01 00:14:42.377838 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.377709    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/089e861c-06db-4d39-bee9-de551706c4c1-cabundle-cert\") pod \"089e861c-06db-4d39-bee9-de551706c4c1\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") "
Aug 01 00:14:42.377838 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.377734    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/089e861c-06db-4d39-bee9-de551706c4c1-kube-rbac-proxy-sar-config\") pod \"089e861c-06db-4d39-bee9-de551706c4c1\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") "
Aug 01 00:14:42.377838 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.377758    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/089e861c-06db-4d39-bee9-de551706c4c1-kserve-provision-location\") pod \"089e861c-06db-4d39-bee9-de551706c4c1\" (UID: \"089e861c-06db-4d39-bee9-de551706c4c1\") "
Aug 01 00:14:42.378148 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.378124    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089e861c-06db-4d39-bee9-de551706c4c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "089e861c-06db-4d39-bee9-de551706c4c1" (UID: "089e861c-06db-4d39-bee9-de551706c4c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:14:42.378148 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.378131    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089e861c-06db-4d39-bee9-de551706c4c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "089e861c-06db-4d39-bee9-de551706c4c1" (UID: "089e861c-06db-4d39-bee9-de551706c4c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:14:42.378757 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.378737    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089e861c-06db-4d39-bee9-de551706c4c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "089e861c-06db-4d39-bee9-de551706c4c1" (UID: "089e861c-06db-4d39-bee9-de551706c4c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:14:42.379803 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.379782    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/089e861c-06db-4d39-bee9-de551706c4c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "089e861c-06db-4d39-bee9-de551706c4c1" (UID: "089e861c-06db-4d39-bee9-de551706c4c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:14:42.379884 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.379848    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089e861c-06db-4d39-bee9-de551706c4c1-kube-api-access-hb9kn" (OuterVolumeSpecName: "kube-api-access-hb9kn") pod "089e861c-06db-4d39-bee9-de551706c4c1" (UID: "089e861c-06db-4d39-bee9-de551706c4c1"). InnerVolumeSpecName "kube-api-access-hb9kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:14:42.478836 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.478758    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb9kn\" (UniqueName: \"kubernetes.io/projected/089e861c-06db-4d39-bee9-de551706c4c1-kube-api-access-hb9kn\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:14:42.478836 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.478787    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/089e861c-06db-4d39-bee9-de551706c4c1-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:14:42.478836 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.478798    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/089e861c-06db-4d39-bee9-de551706c4c1-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:14:42.478836 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.478806    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/089e861c-06db-4d39-bee9-de551706c4c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:14:42.478836 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.478815    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/089e861c-06db-4d39-bee9-de551706c4c1-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:14:42.911255 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.911221    2584 generic.go:356] "Generic (PLEG): container finished" podID="089e861c-06db-4d39-bee9-de551706c4c1" containerID="290993594dd5d81f3ff99903b9521bc03022ae4420ace0d5cb3a0fff077fbe2c" exitCode=0
Aug 01 00:14:42.911407 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.911282    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"
Aug 01 00:14:42.911407 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.911311    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks" event={"ID":"089e861c-06db-4d39-bee9-de551706c4c1","Type":"ContainerDied","Data":"290993594dd5d81f3ff99903b9521bc03022ae4420ace0d5cb3a0fff077fbe2c"}
Aug 01 00:14:42.911407 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.911373    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks" event={"ID":"089e861c-06db-4d39-bee9-de551706c4c1","Type":"ContainerDied","Data":"90404394da200bf1b6287b548aefaa3bb9e63127d8b22926e69895fd04b9494d"}
Aug 01 00:14:42.911407 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.911399    2584 scope.go:117] "RemoveContainer" containerID="7619ee7aa29033026d1b48a3dc82f9e1bf1ce6dc6332c1f4333ac79c62eae7bd"
Aug 01 00:14:42.920123 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.920106    2584 scope.go:117] "RemoveContainer" containerID="290993594dd5d81f3ff99903b9521bc03022ae4420ace0d5cb3a0fff077fbe2c"
Aug 01 00:14:42.927316 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.927295    2584 scope.go:117] "RemoveContainer" containerID="2afe9c3aa61f536d8d894b688f2eb4378882947e2fdff31760321ea698bbd449"
Aug 01 00:14:42.932912 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.932890    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"]
Aug 01 00:14:42.934648 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.934632    2584 scope.go:117] "RemoveContainer" containerID="7619ee7aa29033026d1b48a3dc82f9e1bf1ce6dc6332c1f4333ac79c62eae7bd"
Aug 01 00:14:42.935007 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:14:42.934981    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7619ee7aa29033026d1b48a3dc82f9e1bf1ce6dc6332c1f4333ac79c62eae7bd\": container with ID starting with 7619ee7aa29033026d1b48a3dc82f9e1bf1ce6dc6332c1f4333ac79c62eae7bd not found: ID does not exist" containerID="7619ee7aa29033026d1b48a3dc82f9e1bf1ce6dc6332c1f4333ac79c62eae7bd"
Aug 01 00:14:42.935093 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.935016    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7619ee7aa29033026d1b48a3dc82f9e1bf1ce6dc6332c1f4333ac79c62eae7bd"} err="failed to get container status \"7619ee7aa29033026d1b48a3dc82f9e1bf1ce6dc6332c1f4333ac79c62eae7bd\": rpc error: code = NotFound desc = could not find container \"7619ee7aa29033026d1b48a3dc82f9e1bf1ce6dc6332c1f4333ac79c62eae7bd\": container with ID starting with 7619ee7aa29033026d1b48a3dc82f9e1bf1ce6dc6332c1f4333ac79c62eae7bd not found: ID does not exist"
Aug 01 00:14:42.935093 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.935061    2584 scope.go:117] "RemoveContainer" containerID="290993594dd5d81f3ff99903b9521bc03022ae4420ace0d5cb3a0fff077fbe2c"
Aug 01 00:14:42.935703 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:14:42.935437    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290993594dd5d81f3ff99903b9521bc03022ae4420ace0d5cb3a0fff077fbe2c\": container with ID starting with 290993594dd5d81f3ff99903b9521bc03022ae4420ace0d5cb3a0fff077fbe2c not found: ID does not exist" containerID="290993594dd5d81f3ff99903b9521bc03022ae4420ace0d5cb3a0fff077fbe2c"
Aug 01 00:14:42.935703 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.935466    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290993594dd5d81f3ff99903b9521bc03022ae4420ace0d5cb3a0fff077fbe2c"} err="failed to get container status \"290993594dd5d81f3ff99903b9521bc03022ae4420ace0d5cb3a0fff077fbe2c\": rpc error: code = NotFound desc = could not find container \"290993594dd5d81f3ff99903b9521bc03022ae4420ace0d5cb3a0fff077fbe2c\": container with ID starting with 290993594dd5d81f3ff99903b9521bc03022ae4420ace0d5cb3a0fff077fbe2c not found: ID does not exist"
Aug 01 00:14:42.935703 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.935485    2584 scope.go:117] "RemoveContainer" containerID="2afe9c3aa61f536d8d894b688f2eb4378882947e2fdff31760321ea698bbd449"
Aug 01 00:14:42.935866 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:14:42.935730    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2afe9c3aa61f536d8d894b688f2eb4378882947e2fdff31760321ea698bbd449\": container with ID starting with 2afe9c3aa61f536d8d894b688f2eb4378882947e2fdff31760321ea698bbd449 not found: ID does not exist" containerID="2afe9c3aa61f536d8d894b688f2eb4378882947e2fdff31760321ea698bbd449"
Aug 01 00:14:42.935866 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.935750    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2afe9c3aa61f536d8d894b688f2eb4378882947e2fdff31760321ea698bbd449"} err="failed to get container status \"2afe9c3aa61f536d8d894b688f2eb4378882947e2fdff31760321ea698bbd449\": rpc error: code = NotFound desc = could not find container \"2afe9c3aa61f536d8d894b688f2eb4378882947e2fdff31760321ea698bbd449\": container with ID starting with 2afe9c3aa61f536d8d894b688f2eb4378882947e2fdff31760321ea698bbd449 not found: ID does not exist"
Aug 01 00:14:42.937291 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:42.937271    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-q7dks"]
Aug 01 00:14:43.571139 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:14:43.571099    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="089e861c-06db-4d39-bee9-de551706c4c1" path="/var/lib/kubelet/pods/089e861c-06db-4d39-bee9-de551706c4c1/volumes"
Aug 01 00:16:05.629992 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:16:05.629944    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:16:05.633069 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:16:05.633042    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:17:56.557954 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.557920    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"]
Aug 01 00:17:56.558396 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.558227    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="089e861c-06db-4d39-bee9-de551706c4c1" containerName="kserve-container"
Aug 01 00:17:56.558396 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.558238    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="089e861c-06db-4d39-bee9-de551706c4c1" containerName="kserve-container"
Aug 01 00:17:56.558396 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.558258    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="089e861c-06db-4d39-bee9-de551706c4c1" containerName="storage-initializer"
Aug 01 00:17:56.558396 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.558264    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="089e861c-06db-4d39-bee9-de551706c4c1" containerName="storage-initializer"
Aug 01 00:17:56.558396 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.558271    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="089e861c-06db-4d39-bee9-de551706c4c1" containerName="kube-rbac-proxy"
Aug 01 00:17:56.558396 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.558277    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="089e861c-06db-4d39-bee9-de551706c4c1" containerName="kube-rbac-proxy"
Aug 01 00:17:56.558396 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.558320    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="089e861c-06db-4d39-bee9-de551706c4c1" containerName="kube-rbac-proxy"
Aug 01 00:17:56.558396 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.558327    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="089e861c-06db-4d39-bee9-de551706c4c1" containerName="kserve-container"
Aug 01 00:17:56.561232 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.561213    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.563880 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.563861    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Aug 01 00:17:56.564883 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.564862    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:17:56.565007 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.564895    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:17:56.565007 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.564900    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:17:56.565323 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.565179    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:17:56.565323 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.565208    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:17:56.565323 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.565260    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Aug 01 00:17:56.570450 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.570425    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"]
Aug 01 00:17:56.659212 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.659174    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.659384 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.659221    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f31e243-04b8-4bfc-b599-2d2f8c9624db-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.659384 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.659304    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f31e243-04b8-4bfc-b599-2d2f8c9624db-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.659384 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.659354    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj5bm\" (UniqueName: \"kubernetes.io/projected/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kube-api-access-tj5bm\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.659503 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.659400    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.760763 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.760722    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.761005 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.760783    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.761005 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.760809    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f31e243-04b8-4bfc-b599-2d2f8c9624db-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.761005 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.760838    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f31e243-04b8-4bfc-b599-2d2f8c9624db-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.761005 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.760875    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj5bm\" (UniqueName: \"kubernetes.io/projected/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kube-api-access-tj5bm\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.761249 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.761188    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.761533 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.761511    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.761612 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.761535    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f31e243-04b8-4bfc-b599-2d2f8c9624db-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.763227 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.763209    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f31e243-04b8-4bfc-b599-2d2f8c9624db-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.769491 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.769467    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj5bm\" (UniqueName: \"kubernetes.io/projected/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kube-api-access-tj5bm\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.872803 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.872714    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:17:56.998543 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:56.998518    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"]
Aug 01 00:17:57.000142 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:17:57.000114    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f31e243_04b8_4bfc_b599_2d2f8c9624db.slice/crio-9b594ac25212cb5f6355220aff02d717148b7cc9f552e3f60abebbb320f1b334 WatchSource:0}: Error finding container 9b594ac25212cb5f6355220aff02d717148b7cc9f552e3f60abebbb320f1b334: Status 404 returned error can't find the container with id 9b594ac25212cb5f6355220aff02d717148b7cc9f552e3f60abebbb320f1b334
Aug 01 00:17:57.711168 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:57.711110    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" event={"ID":"4f31e243-04b8-4bfc-b599-2d2f8c9624db","Type":"ContainerStarted","Data":"25294eeadaef3502f580354df136a2db61a55e71703b954fecfa24ed92b1f2fd"}
Aug 01 00:17:57.711568 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:17:57.711218    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" event={"ID":"4f31e243-04b8-4bfc-b599-2d2f8c9624db","Type":"ContainerStarted","Data":"9b594ac25212cb5f6355220aff02d717148b7cc9f552e3f60abebbb320f1b334"}
Aug 01 00:18:00.725121 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:00.725085    2584 generic.go:356] "Generic (PLEG): container finished" podID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerID="25294eeadaef3502f580354df136a2db61a55e71703b954fecfa24ed92b1f2fd" exitCode=0
Aug 01 00:18:00.725456 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:00.725155    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" event={"ID":"4f31e243-04b8-4bfc-b599-2d2f8c9624db","Type":"ContainerDied","Data":"25294eeadaef3502f580354df136a2db61a55e71703b954fecfa24ed92b1f2fd"}
Aug 01 00:18:00.726288 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:00.726274    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:18:14.804110 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:14.804071    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" event={"ID":"4f31e243-04b8-4bfc-b599-2d2f8c9624db","Type":"ContainerStarted","Data":"4432f8898bf3b1f40c0c73f2eb9c2f6ef6c0008b138bdd915c6b0b3617226155"}
Aug 01 00:18:14.804559 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:14.804142    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" event={"ID":"4f31e243-04b8-4bfc-b599-2d2f8c9624db","Type":"ContainerStarted","Data":"1d10e7e2fd21d25ec33f1789115bcd9b78d9f891d4c802b3a3e5adea19a0ad8c"}
Aug 01 00:18:14.804559 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:14.804429    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:18:14.804559 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:14.804481    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:18:14.805932 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:14.805904    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 01 00:18:14.826636 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:14.826589    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" podStartSLOduration=2.137520934 podStartE2EDuration="18.82657339s" totalImagesPullingTime="13.088143063s" totalInitContainerRuntime="3.600909393s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:18:14.824240309 +0000 UTC m=+1929.697883897" watchObservedRunningTime="2026-08-01 00:18:14.82657339 +0000 UTC m=+1929.700217000"
Aug 01 00:18:15.808442 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:15.808400    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 01 00:18:20.813115 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:20.813082    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:18:20.813673 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:20.813644    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 01 00:18:30.813585 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:30.813544    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 01 00:18:40.814183 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:40.814146    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 01 00:18:50.813920 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:50.813878    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 01 00:18:51.254066 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.253983    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"]
Aug 01 00:18:51.290273 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.290241    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"]
Aug 01 00:18:51.290430 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.290366    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:51.293183 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.293160    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Aug 01 00:18:51.293183 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.293176    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Aug 01 00:18:51.409193 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.409159    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c071582-9874-4b30-b193-194ff29b3b0f-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:51.409406 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.409203    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c071582-9874-4b30-b193-194ff29b3b0f-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:51.409406 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.409226    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2ll\" (UniqueName: \"kubernetes.io/projected/8c071582-9874-4b30-b193-194ff29b3b0f-kube-api-access-xk2ll\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:51.409406 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.409243    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c071582-9874-4b30-b193-194ff29b3b0f-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:51.409406 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.409329    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c071582-9874-4b30-b193-194ff29b3b0f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:51.510513 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.510418    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c071582-9874-4b30-b193-194ff29b3b0f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:51.510513 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.510481    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c071582-9874-4b30-b193-194ff29b3b0f-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:51.510513 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.510511    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c071582-9874-4b30-b193-194ff29b3b0f-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:51.510778 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.510532    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2ll\" (UniqueName: \"kubernetes.io/projected/8c071582-9874-4b30-b193-194ff29b3b0f-kube-api-access-xk2ll\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:51.510778 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.510551    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c071582-9874-4b30-b193-194ff29b3b0f-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:51.510778 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:18:51.510660    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Aug 01 00:18:51.510778 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:18:51.510721    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c071582-9874-4b30-b193-194ff29b3b0f-proxy-tls podName:8c071582-9874-4b30-b193-194ff29b3b0f nodeName:}" failed. No retries permitted until 2026-08-01 00:18:52.010700034 +0000 UTC m=+1966.884343640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c071582-9874-4b30-b193-194ff29b3b0f-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" (UID: "8c071582-9874-4b30-b193-194ff29b3b0f") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Aug 01 00:18:51.511026 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.511009    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c071582-9874-4b30-b193-194ff29b3b0f-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:51.511181 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.511163    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c071582-9874-4b30-b193-194ff29b3b0f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:51.511223 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.511195    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c071582-9874-4b30-b193-194ff29b3b0f-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:51.519042 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:51.519011    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2ll\" (UniqueName: \"kubernetes.io/projected/8c071582-9874-4b30-b193-194ff29b3b0f-kube-api-access-xk2ll\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:52.014818 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:52.014780    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c071582-9874-4b30-b193-194ff29b3b0f-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:52.017363 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:52.017336    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c071582-9874-4b30-b193-194ff29b3b0f-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:52.201370 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:52.201330    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:52.331367 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:52.331341    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"]
Aug 01 00:18:52.332858 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:18:52.332823    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c071582_9874_4b30_b193_194ff29b3b0f.slice/crio-56d404c5c662470a11440091ecb6bc4c4cb661331c42094184f0cd1cc938965a WatchSource:0}: Error finding container 56d404c5c662470a11440091ecb6bc4c4cb661331c42094184f0cd1cc938965a: Status 404 returned error can't find the container with id 56d404c5c662470a11440091ecb6bc4c4cb661331c42094184f0cd1cc938965a
Aug 01 00:18:52.966628 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:52.966596    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" event={"ID":"8c071582-9874-4b30-b193-194ff29b3b0f","Type":"ContainerStarted","Data":"28a8a5ac1e6696f1ca0df9bdd2b427b124c12b27377c8f2b110a1c0331b476c6"}
Aug 01 00:18:52.966807 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:52.966649    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" event={"ID":"8c071582-9874-4b30-b193-194ff29b3b0f","Type":"ContainerStarted","Data":"56d404c5c662470a11440091ecb6bc4c4cb661331c42094184f0cd1cc938965a"}
Aug 01 00:18:56.984366 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:56.984327    2584 generic.go:356] "Generic (PLEG): container finished" podID="8c071582-9874-4b30-b193-194ff29b3b0f" containerID="28a8a5ac1e6696f1ca0df9bdd2b427b124c12b27377c8f2b110a1c0331b476c6" exitCode=0
Aug 01 00:18:56.984794 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:56.984401    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" event={"ID":"8c071582-9874-4b30-b193-194ff29b3b0f","Type":"ContainerDied","Data":"28a8a5ac1e6696f1ca0df9bdd2b427b124c12b27377c8f2b110a1c0331b476c6"}
Aug 01 00:18:57.991149 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:57.991069    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" event={"ID":"8c071582-9874-4b30-b193-194ff29b3b0f","Type":"ContainerStarted","Data":"8af6133ea98b8b9a585997619a7a9ede887d3d1e0432f816e51f1456ca9953fc"}
Aug 01 00:18:57.991149 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:57.991127    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" event={"ID":"8c071582-9874-4b30-b193-194ff29b3b0f","Type":"ContainerStarted","Data":"b0cef83e3b7f20caee5f18a036053d602c0a59c62452c9502e76175b13ad55fe"}
Aug 01 00:18:57.991613 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:57.991400    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:57.991613 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:57.991535    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:18:57.992761 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:57.992735    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 01 00:18:58.013594 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:58.013548    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" podStartSLOduration=3.263508675 podStartE2EDuration="7.013532269s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.750023594s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:18:58.011456178 +0000 UTC m=+1972.885099788" watchObservedRunningTime="2026-08-01 00:18:58.013532269 +0000 UTC m=+1972.887175879"
Aug 01 00:18:58.995393 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:18:58.995355    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 01 00:19:00.813780 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:00.813729    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 01 00:19:04.000273 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:04.000242    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:19:04.000715 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:04.000690    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 01 00:19:10.813852 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:10.813811    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 01 00:19:14.001419 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:14.001375    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 01 00:19:20.814126 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:20.814090    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 01 00:19:24.001460 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:24.001425    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 01 00:19:30.814116 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:30.814084    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:19:34.000800 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:34.000764    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 01 00:19:36.750067 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:36.750028    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"]
Aug 01 00:19:36.750538 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:36.750271    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kserve-container" containerID="cri-o://1d10e7e2fd21d25ec33f1789115bcd9b78d9f891d4c802b3a3e5adea19a0ad8c" gracePeriod=30
Aug 01 00:19:36.750538 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:36.750289    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kube-rbac-proxy" containerID="cri-o://4432f8898bf3b1f40c0c73f2eb9c2f6ef6c0008b138bdd915c6b0b3617226155" gracePeriod=30
Aug 01 00:19:37.170341 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:37.170309    2584 generic.go:356] "Generic (PLEG): container finished" podID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerID="4432f8898bf3b1f40c0c73f2eb9c2f6ef6c0008b138bdd915c6b0b3617226155" exitCode=2
Aug 01 00:19:37.170515 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:37.170361    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" event={"ID":"4f31e243-04b8-4bfc-b599-2d2f8c9624db","Type":"ContainerDied","Data":"4432f8898bf3b1f40c0c73f2eb9c2f6ef6c0008b138bdd915c6b0b3617226155"}
Aug 01 00:19:40.808893 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:40.808845    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Aug 01 00:19:40.813898 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:40.813872    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 01 00:19:41.495715 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.495692    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:19:41.522163 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.522127    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kube-rbac-proxy-sar-config\") pod \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") "
Aug 01 00:19:41.522338 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.522174    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kserve-provision-location\") pod \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") "
Aug 01 00:19:41.522338 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.522195    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f31e243-04b8-4bfc-b599-2d2f8c9624db-cabundle-cert\") pod \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") "
Aug 01 00:19:41.522338 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.522227    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj5bm\" (UniqueName: \"kubernetes.io/projected/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kube-api-access-tj5bm\") pod \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") "
Aug 01 00:19:41.522338 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.522292    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f31e243-04b8-4bfc-b599-2d2f8c9624db-proxy-tls\") pod \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\" (UID: \"4f31e243-04b8-4bfc-b599-2d2f8c9624db\") "
Aug 01 00:19:41.522575 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.522523    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f31e243-04b8-4bfc-b599-2d2f8c9624db" (UID: "4f31e243-04b8-4bfc-b599-2d2f8c9624db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:19:41.522575 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.522535    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f31e243-04b8-4bfc-b599-2d2f8c9624db" (UID: "4f31e243-04b8-4bfc-b599-2d2f8c9624db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:19:41.522887 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.522854    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f31e243-04b8-4bfc-b599-2d2f8c9624db-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f31e243-04b8-4bfc-b599-2d2f8c9624db" (UID: "4f31e243-04b8-4bfc-b599-2d2f8c9624db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:19:41.524517 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.524480    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f31e243-04b8-4bfc-b599-2d2f8c9624db-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f31e243-04b8-4bfc-b599-2d2f8c9624db" (UID: "4f31e243-04b8-4bfc-b599-2d2f8c9624db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:19:41.524660 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.524593    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kube-api-access-tj5bm" (OuterVolumeSpecName: "kube-api-access-tj5bm") pod "4f31e243-04b8-4bfc-b599-2d2f8c9624db" (UID: "4f31e243-04b8-4bfc-b599-2d2f8c9624db"). InnerVolumeSpecName "kube-api-access-tj5bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:19:41.623458 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.623427    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f31e243-04b8-4bfc-b599-2d2f8c9624db-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:19:41.623458 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.623453    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:19:41.623458 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.623463    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:19:41.623667 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.623473    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f31e243-04b8-4bfc-b599-2d2f8c9624db-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:19:41.623667 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:41.623481    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj5bm\" (UniqueName: \"kubernetes.io/projected/4f31e243-04b8-4bfc-b599-2d2f8c9624db-kube-api-access-tj5bm\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:19:42.193316 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:42.193284    2584 generic.go:356] "Generic (PLEG): container finished" podID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerID="1d10e7e2fd21d25ec33f1789115bcd9b78d9f891d4c802b3a3e5adea19a0ad8c" exitCode=0
Aug 01 00:19:42.193740 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:42.193322    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" event={"ID":"4f31e243-04b8-4bfc-b599-2d2f8c9624db","Type":"ContainerDied","Data":"1d10e7e2fd21d25ec33f1789115bcd9b78d9f891d4c802b3a3e5adea19a0ad8c"}
Aug 01 00:19:42.193740 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:42.193356    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"
Aug 01 00:19:42.193740 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:42.193371    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc" event={"ID":"4f31e243-04b8-4bfc-b599-2d2f8c9624db","Type":"ContainerDied","Data":"9b594ac25212cb5f6355220aff02d717148b7cc9f552e3f60abebbb320f1b334"}
Aug 01 00:19:42.193740 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:42.193397    2584 scope.go:117] "RemoveContainer" containerID="4432f8898bf3b1f40c0c73f2eb9c2f6ef6c0008b138bdd915c6b0b3617226155"
Aug 01 00:19:42.202000 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:42.201944    2584 scope.go:117] "RemoveContainer" containerID="1d10e7e2fd21d25ec33f1789115bcd9b78d9f891d4c802b3a3e5adea19a0ad8c"
Aug 01 00:19:42.210510 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:42.210489    2584 scope.go:117] "RemoveContainer" containerID="25294eeadaef3502f580354df136a2db61a55e71703b954fecfa24ed92b1f2fd"
Aug 01 00:19:42.212066 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:42.212043    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"]
Aug 01 00:19:42.216377 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:42.216353    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-kqhbc"]
Aug 01 00:19:42.218734 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:42.218712    2584 scope.go:117] "RemoveContainer" containerID="4432f8898bf3b1f40c0c73f2eb9c2f6ef6c0008b138bdd915c6b0b3617226155"
Aug 01 00:19:42.219075 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:19:42.219048    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4432f8898bf3b1f40c0c73f2eb9c2f6ef6c0008b138bdd915c6b0b3617226155\": container with ID starting with 4432f8898bf3b1f40c0c73f2eb9c2f6ef6c0008b138bdd915c6b0b3617226155 not found: ID does not exist" containerID="4432f8898bf3b1f40c0c73f2eb9c2f6ef6c0008b138bdd915c6b0b3617226155"
Aug 01 00:19:42.219145 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:42.219082    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4432f8898bf3b1f40c0c73f2eb9c2f6ef6c0008b138bdd915c6b0b3617226155"} err="failed to get container status \"4432f8898bf3b1f40c0c73f2eb9c2f6ef6c0008b138bdd915c6b0b3617226155\": rpc error: code = NotFound desc = could not find container \"4432f8898bf3b1f40c0c73f2eb9c2f6ef6c0008b138bdd915c6b0b3617226155\": container with ID starting with 4432f8898bf3b1f40c0c73f2eb9c2f6ef6c0008b138bdd915c6b0b3617226155 not found: ID does not exist"
Aug 01 00:19:42.219145 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:42.219102    2584 scope.go:117] "RemoveContainer" containerID="1d10e7e2fd21d25ec33f1789115bcd9b78d9f891d4c802b3a3e5adea19a0ad8c"
Aug 01 00:19:42.219363 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:19:42.219345    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d10e7e2fd21d25ec33f1789115bcd9b78d9f891d4c802b3a3e5adea19a0ad8c\": container with ID starting with 1d10e7e2fd21d25ec33f1789115bcd9b78d9f891d4c802b3a3e5adea19a0ad8c not found: ID does not exist" containerID="1d10e7e2fd21d25ec33f1789115bcd9b78d9f891d4c802b3a3e5adea19a0ad8c"
Aug 01 00:19:42.219412 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:42.219376    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d10e7e2fd21d25ec33f1789115bcd9b78d9f891d4c802b3a3e5adea19a0ad8c"} err="failed to get container status \"1d10e7e2fd21d25ec33f1789115bcd9b78d9f891d4c802b3a3e5adea19a0ad8c\": rpc error: code = NotFound desc = could not find container \"1d10e7e2fd21d25ec33f1789115bcd9b78d9f891d4c802b3a3e5adea19a0ad8c\": container with ID starting with 1d10e7e2fd21d25ec33f1789115bcd9b78d9f891d4c802b3a3e5adea19a0ad8c not found: ID does not exist"
Aug 01 00:19:42.219412 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:42.219394    2584 scope.go:117] "RemoveContainer" containerID="25294eeadaef3502f580354df136a2db61a55e71703b954fecfa24ed92b1f2fd"
Aug 01 00:19:42.219637 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:19:42.219619    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25294eeadaef3502f580354df136a2db61a55e71703b954fecfa24ed92b1f2fd\": container with ID starting with 25294eeadaef3502f580354df136a2db61a55e71703b954fecfa24ed92b1f2fd not found: ID does not exist" containerID="25294eeadaef3502f580354df136a2db61a55e71703b954fecfa24ed92b1f2fd"
Aug 01 00:19:42.219692 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:42.219644    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25294eeadaef3502f580354df136a2db61a55e71703b954fecfa24ed92b1f2fd"} err="failed to get container status \"25294eeadaef3502f580354df136a2db61a55e71703b954fecfa24ed92b1f2fd\": rpc error: code = NotFound desc = could not find container \"25294eeadaef3502f580354df136a2db61a55e71703b954fecfa24ed92b1f2fd\": container with ID starting with 25294eeadaef3502f580354df136a2db61a55e71703b954fecfa24ed92b1f2fd not found: ID does not exist"
Aug 01 00:19:43.572049 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:43.572013    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" path="/var/lib/kubelet/pods/4f31e243-04b8-4bfc-b599-2d2f8c9624db/volumes"
Aug 01 00:19:44.001720 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:44.001636    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 01 00:19:54.001494 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:19:54.001456    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 01 00:20:04.001404 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:04.001363    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 01 00:20:06.565834 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:06.565794    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 01 00:20:16.566142 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:16.566108    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:20:21.352726 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:21.352686    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"]
Aug 01 00:20:21.353188 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:21.352995    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kserve-container" containerID="cri-o://b0cef83e3b7f20caee5f18a036053d602c0a59c62452c9502e76175b13ad55fe" gracePeriod=30
Aug 01 00:20:21.353188 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:21.353046    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kube-rbac-proxy" containerID="cri-o://8af6133ea98b8b9a585997619a7a9ede887d3d1e0432f816e51f1456ca9953fc" gracePeriod=30
Aug 01 00:20:22.369217 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:22.369184    2584 generic.go:356] "Generic (PLEG): container finished" podID="8c071582-9874-4b30-b193-194ff29b3b0f" containerID="8af6133ea98b8b9a585997619a7a9ede887d3d1e0432f816e51f1456ca9953fc" exitCode=2
Aug 01 00:20:22.369626 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:22.369268    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" event={"ID":"8c071582-9874-4b30-b193-194ff29b3b0f","Type":"ContainerDied","Data":"8af6133ea98b8b9a585997619a7a9ede887d3d1e0432f816e51f1456ca9953fc"}
Aug 01 00:20:23.996123 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:23.996082    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Aug 01 00:20:25.804996 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:25.804948    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:20:25.901667 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:25.901586    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk2ll\" (UniqueName: \"kubernetes.io/projected/8c071582-9874-4b30-b193-194ff29b3b0f-kube-api-access-xk2ll\") pod \"8c071582-9874-4b30-b193-194ff29b3b0f\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") "
Aug 01 00:20:25.901667 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:25.901656    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c071582-9874-4b30-b193-194ff29b3b0f-kube-rbac-proxy-sar-config\") pod \"8c071582-9874-4b30-b193-194ff29b3b0f\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") "
Aug 01 00:20:25.901876 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:25.901682    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c071582-9874-4b30-b193-194ff29b3b0f-cabundle-cert\") pod \"8c071582-9874-4b30-b193-194ff29b3b0f\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") "
Aug 01 00:20:25.901876 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:25.901731    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c071582-9874-4b30-b193-194ff29b3b0f-kserve-provision-location\") pod \"8c071582-9874-4b30-b193-194ff29b3b0f\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") "
Aug 01 00:20:25.901876 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:25.901759    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c071582-9874-4b30-b193-194ff29b3b0f-proxy-tls\") pod \"8c071582-9874-4b30-b193-194ff29b3b0f\" (UID: \"8c071582-9874-4b30-b193-194ff29b3b0f\") "
Aug 01 00:20:25.902076 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:25.902055    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c071582-9874-4b30-b193-194ff29b3b0f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c071582-9874-4b30-b193-194ff29b3b0f" (UID: "8c071582-9874-4b30-b193-194ff29b3b0f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:20:25.902125 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:25.902069    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c071582-9874-4b30-b193-194ff29b3b0f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c071582-9874-4b30-b193-194ff29b3b0f" (UID: "8c071582-9874-4b30-b193-194ff29b3b0f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:20:25.902171 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:25.902108    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c071582-9874-4b30-b193-194ff29b3b0f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c071582-9874-4b30-b193-194ff29b3b0f" (UID: "8c071582-9874-4b30-b193-194ff29b3b0f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:20:25.903763 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:25.903741    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c071582-9874-4b30-b193-194ff29b3b0f-kube-api-access-xk2ll" (OuterVolumeSpecName: "kube-api-access-xk2ll") pod "8c071582-9874-4b30-b193-194ff29b3b0f" (UID: "8c071582-9874-4b30-b193-194ff29b3b0f"). InnerVolumeSpecName "kube-api-access-xk2ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:20:25.903820 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:25.903758    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c071582-9874-4b30-b193-194ff29b3b0f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c071582-9874-4b30-b193-194ff29b3b0f" (UID: "8c071582-9874-4b30-b193-194ff29b3b0f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:20:26.003161 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.003113    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c071582-9874-4b30-b193-194ff29b3b0f-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:20:26.003161 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.003153    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c071582-9874-4b30-b193-194ff29b3b0f-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:20:26.003161 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.003164    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk2ll\" (UniqueName: \"kubernetes.io/projected/8c071582-9874-4b30-b193-194ff29b3b0f-kube-api-access-xk2ll\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:20:26.003161 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.003172    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c071582-9874-4b30-b193-194ff29b3b0f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:20:26.003417 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.003181    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c071582-9874-4b30-b193-194ff29b3b0f-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:20:26.388030 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.387996    2584 generic.go:356] "Generic (PLEG): container finished" podID="8c071582-9874-4b30-b193-194ff29b3b0f" containerID="b0cef83e3b7f20caee5f18a036053d602c0a59c62452c9502e76175b13ad55fe" exitCode=0
Aug 01 00:20:26.388187 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.388052    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"
Aug 01 00:20:26.388187 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.388079    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" event={"ID":"8c071582-9874-4b30-b193-194ff29b3b0f","Type":"ContainerDied","Data":"b0cef83e3b7f20caee5f18a036053d602c0a59c62452c9502e76175b13ad55fe"}
Aug 01 00:20:26.388187 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.388138    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt" event={"ID":"8c071582-9874-4b30-b193-194ff29b3b0f","Type":"ContainerDied","Data":"56d404c5c662470a11440091ecb6bc4c4cb661331c42094184f0cd1cc938965a"}
Aug 01 00:20:26.388187 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.388168    2584 scope.go:117] "RemoveContainer" containerID="8af6133ea98b8b9a585997619a7a9ede887d3d1e0432f816e51f1456ca9953fc"
Aug 01 00:20:26.397198 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.397170    2584 scope.go:117] "RemoveContainer" containerID="b0cef83e3b7f20caee5f18a036053d602c0a59c62452c9502e76175b13ad55fe"
Aug 01 00:20:26.404706 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.404687    2584 scope.go:117] "RemoveContainer" containerID="28a8a5ac1e6696f1ca0df9bdd2b427b124c12b27377c8f2b110a1c0331b476c6"
Aug 01 00:20:26.412725 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.412702    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"]
Aug 01 00:20:26.413274 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.413258    2584 scope.go:117] "RemoveContainer" containerID="8af6133ea98b8b9a585997619a7a9ede887d3d1e0432f816e51f1456ca9953fc"
Aug 01 00:20:26.413561 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:20:26.413538    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af6133ea98b8b9a585997619a7a9ede887d3d1e0432f816e51f1456ca9953fc\": container with ID starting with 8af6133ea98b8b9a585997619a7a9ede887d3d1e0432f816e51f1456ca9953fc not found: ID does not exist" containerID="8af6133ea98b8b9a585997619a7a9ede887d3d1e0432f816e51f1456ca9953fc"
Aug 01 00:20:26.413624 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.413565    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af6133ea98b8b9a585997619a7a9ede887d3d1e0432f816e51f1456ca9953fc"} err="failed to get container status \"8af6133ea98b8b9a585997619a7a9ede887d3d1e0432f816e51f1456ca9953fc\": rpc error: code = NotFound desc = could not find container \"8af6133ea98b8b9a585997619a7a9ede887d3d1e0432f816e51f1456ca9953fc\": container with ID starting with 8af6133ea98b8b9a585997619a7a9ede887d3d1e0432f816e51f1456ca9953fc not found: ID does not exist"
Aug 01 00:20:26.413624 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.413588    2584 scope.go:117] "RemoveContainer" containerID="b0cef83e3b7f20caee5f18a036053d602c0a59c62452c9502e76175b13ad55fe"
Aug 01 00:20:26.413885 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:20:26.413845    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0cef83e3b7f20caee5f18a036053d602c0a59c62452c9502e76175b13ad55fe\": container with ID starting with b0cef83e3b7f20caee5f18a036053d602c0a59c62452c9502e76175b13ad55fe not found: ID does not exist" containerID="b0cef83e3b7f20caee5f18a036053d602c0a59c62452c9502e76175b13ad55fe"
Aug 01 00:20:26.414020 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.413889    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0cef83e3b7f20caee5f18a036053d602c0a59c62452c9502e76175b13ad55fe"} err="failed to get container status \"b0cef83e3b7f20caee5f18a036053d602c0a59c62452c9502e76175b13ad55fe\": rpc error: code = NotFound desc = could not find container \"b0cef83e3b7f20caee5f18a036053d602c0a59c62452c9502e76175b13ad55fe\": container with ID starting with b0cef83e3b7f20caee5f18a036053d602c0a59c62452c9502e76175b13ad55fe not found: ID does not exist"
Aug 01 00:20:26.414020 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.413920    2584 scope.go:117] "RemoveContainer" containerID="28a8a5ac1e6696f1ca0df9bdd2b427b124c12b27377c8f2b110a1c0331b476c6"
Aug 01 00:20:26.414248 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:20:26.414219    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a8a5ac1e6696f1ca0df9bdd2b427b124c12b27377c8f2b110a1c0331b476c6\": container with ID starting with 28a8a5ac1e6696f1ca0df9bdd2b427b124c12b27377c8f2b110a1c0331b476c6 not found: ID does not exist" containerID="28a8a5ac1e6696f1ca0df9bdd2b427b124c12b27377c8f2b110a1c0331b476c6"
Aug 01 00:20:26.414336 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.414243    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a8a5ac1e6696f1ca0df9bdd2b427b124c12b27377c8f2b110a1c0331b476c6"} err="failed to get container status \"28a8a5ac1e6696f1ca0df9bdd2b427b124c12b27377c8f2b110a1c0331b476c6\": rpc error: code = NotFound desc = could not find container \"28a8a5ac1e6696f1ca0df9bdd2b427b124c12b27377c8f2b110a1c0331b476c6\": container with ID starting with 28a8a5ac1e6696f1ca0df9bdd2b427b124c12b27377c8f2b110a1c0331b476c6 not found: ID does not exist"
Aug 01 00:20:26.418761 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.418741    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbc65f9f-hckjt"]
Aug 01 00:20:26.545585 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.545553    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"]
Aug 01 00:20:26.545893 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.545879    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="storage-initializer"
Aug 01 00:20:26.545938 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.545895    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="storage-initializer"
Aug 01 00:20:26.545938 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.545905    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kserve-container"
Aug 01 00:20:26.545938 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.545911    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kserve-container"
Aug 01 00:20:26.545938 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.545923    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="storage-initializer"
Aug 01 00:20:26.545938 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.545929    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="storage-initializer"
Aug 01 00:20:26.545938 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.545937    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kube-rbac-proxy"
Aug 01 00:20:26.546201 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.545942    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kube-rbac-proxy"
Aug 01 00:20:26.546201 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.545948    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kserve-container"
Aug 01 00:20:26.546201 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.545954    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kserve-container"
Aug 01 00:20:26.546201 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.545979    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kube-rbac-proxy"
Aug 01 00:20:26.546201 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.545984    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kube-rbac-proxy"
Aug 01 00:20:26.546201 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.546032    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kube-rbac-proxy"
Aug 01 00:20:26.546201 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.546040    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f31e243-04b8-4bfc-b599-2d2f8c9624db" containerName="kserve-container"
Aug 01 00:20:26.546201 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.546046    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kserve-container"
Aug 01 00:20:26.546201 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.546051    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" containerName="kube-rbac-proxy"
Aug 01 00:20:26.550739 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.550720    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.553404 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.553378    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:20:26.553553 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.553529    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:20:26.553639 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.553508    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:20:26.553639 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.553571    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:20:26.553639 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.553599    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Aug 01 00:20:26.553803 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.553644    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:20:26.553803 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.553723    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Aug 01 00:20:26.559094 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.559075    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"]
Aug 01 00:20:26.607242 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.607219    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aee6e647-f05f-4903-8405-94a54f62e273-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-5l76g\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.607378 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.607253    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aee6e647-f05f-4903-8405-94a54f62e273-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-5l76g\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.607378 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.607289    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aee6e647-f05f-4903-8405-94a54f62e273-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-5l76g\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.607378 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.607358    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x64wv\" (UniqueName: \"kubernetes.io/projected/aee6e647-f05f-4903-8405-94a54f62e273-kube-api-access-x64wv\") pod \"isvc-tensorflow-predictor-547dfc5669-5l76g\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.607496 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.607392    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aee6e647-f05f-4903-8405-94a54f62e273-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-5l76g\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.708729 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.708648    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aee6e647-f05f-4903-8405-94a54f62e273-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-5l76g\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.708729 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.708692    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x64wv\" (UniqueName: \"kubernetes.io/projected/aee6e647-f05f-4903-8405-94a54f62e273-kube-api-access-x64wv\") pod \"isvc-tensorflow-predictor-547dfc5669-5l76g\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.708729 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.708727    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aee6e647-f05f-4903-8405-94a54f62e273-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-5l76g\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.709021 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.708777    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aee6e647-f05f-4903-8405-94a54f62e273-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-5l76g\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.709021 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.708809    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aee6e647-f05f-4903-8405-94a54f62e273-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-5l76g\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.709240 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.709220    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aee6e647-f05f-4903-8405-94a54f62e273-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-5l76g\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.709556 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.709535    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aee6e647-f05f-4903-8405-94a54f62e273-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-5l76g\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.709636 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.709555    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aee6e647-f05f-4903-8405-94a54f62e273-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-5l76g\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.711264 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.711245    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aee6e647-f05f-4903-8405-94a54f62e273-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-5l76g\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.716549 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.716528    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x64wv\" (UniqueName: \"kubernetes.io/projected/aee6e647-f05f-4903-8405-94a54f62e273-kube-api-access-x64wv\") pod \"isvc-tensorflow-predictor-547dfc5669-5l76g\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.862341 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.862303    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:26.985014 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:26.984990    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"]
Aug 01 00:20:26.986989 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:20:26.986929    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee6e647_f05f_4903_8405_94a54f62e273.slice/crio-bf63959ab96312b3a57c4ffbe5e734e3218f9345b685795f46a6768a375f79e2 WatchSource:0}: Error finding container bf63959ab96312b3a57c4ffbe5e734e3218f9345b685795f46a6768a375f79e2: Status 404 returned error can't find the container with id bf63959ab96312b3a57c4ffbe5e734e3218f9345b685795f46a6768a375f79e2
Aug 01 00:20:27.396116 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:27.396032    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" event={"ID":"aee6e647-f05f-4903-8405-94a54f62e273","Type":"ContainerStarted","Data":"8e5010bf3a0f9953c046955dfb64eb4bd236f094bcb5f2e3d4a635a72d32d842"}
Aug 01 00:20:27.396116 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:27.396098    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" event={"ID":"aee6e647-f05f-4903-8405-94a54f62e273","Type":"ContainerStarted","Data":"bf63959ab96312b3a57c4ffbe5e734e3218f9345b685795f46a6768a375f79e2"}
Aug 01 00:20:27.570385 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:27.570355    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c071582-9874-4b30-b193-194ff29b3b0f" path="/var/lib/kubelet/pods/8c071582-9874-4b30-b193-194ff29b3b0f/volumes"
Aug 01 00:20:32.417601 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:32.417564    2584 generic.go:356] "Generic (PLEG): container finished" podID="aee6e647-f05f-4903-8405-94a54f62e273" containerID="8e5010bf3a0f9953c046955dfb64eb4bd236f094bcb5f2e3d4a635a72d32d842" exitCode=0
Aug 01 00:20:32.418063 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:32.417639    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" event={"ID":"aee6e647-f05f-4903-8405-94a54f62e273","Type":"ContainerDied","Data":"8e5010bf3a0f9953c046955dfb64eb4bd236f094bcb5f2e3d4a635a72d32d842"}
Aug 01 00:20:36.439602 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:36.439564    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" event={"ID":"aee6e647-f05f-4903-8405-94a54f62e273","Type":"ContainerStarted","Data":"3ce377673568ba2c0d3997c61b08e763c708f813a27cd6ddc293687c9525e5c4"}
Aug 01 00:20:36.440072 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:36.439660    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" event={"ID":"aee6e647-f05f-4903-8405-94a54f62e273","Type":"ContainerStarted","Data":"c133d0da510e340496db947c7468c46b01a15fcb2599b04490c68976a2f89dfd"}
Aug 01 00:20:36.440072 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:36.439997    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:36.440183 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:36.440133    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:36.441631 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:36.441605    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 01 00:20:36.459534 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:36.459451    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" podStartSLOduration=2.215324647 podStartE2EDuration="10.459437839s" totalImagesPullingTime="3.788339942s" totalInitContainerRuntime="4.45577325s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:20:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:20:36.457056533 +0000 UTC m=+2071.330700144" watchObservedRunningTime="2026-08-01 00:20:36.459437839 +0000 UTC m=+2071.333081448"
Aug 01 00:20:37.444368 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:37.444333    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 01 00:20:42.448543 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:42.448511    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:20:42.449028 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:42.449001    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 01 00:20:52.449873 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:20:52.449839    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:21:05.653993 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:05.653948    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:21:05.658251 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:05.658227    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:21:07.349291 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:07.349256    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"]
Aug 01 00:21:07.349680 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:07.349551    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kserve-container" containerID="cri-o://c133d0da510e340496db947c7468c46b01a15fcb2599b04490c68976a2f89dfd" gracePeriod=30
Aug 01 00:21:07.349680 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:07.349581    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kube-rbac-proxy" containerID="cri-o://3ce377673568ba2c0d3997c61b08e763c708f813a27cd6ddc293687c9525e5c4" gracePeriod=30
Aug 01 00:21:07.444895 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:07.444845    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Aug 01 00:21:07.573801 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:07.573770    2584 generic.go:356] "Generic (PLEG): container finished" podID="aee6e647-f05f-4903-8405-94a54f62e273" containerID="3ce377673568ba2c0d3997c61b08e763c708f813a27cd6ddc293687c9525e5c4" exitCode=2
Aug 01 00:21:07.573991 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:07.573809    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" event={"ID":"aee6e647-f05f-4903-8405-94a54f62e273","Type":"ContainerDied","Data":"3ce377673568ba2c0d3997c61b08e763c708f813a27cd6ddc293687c9525e5c4"}
Aug 01 00:21:11.971680 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:11.971642    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"]
Aug 01 00:21:11.974039 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:11.974016    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:11.976630 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:11.976610    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Aug 01 00:21:11.976749 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:11.976729    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Aug 01 00:21:11.983897 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:11.983876    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"]
Aug 01 00:21:11.991729 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:11.991701    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5113f39-2584-4bd0-a0e2-fa509bb104d7-proxy-tls\") pod \"isvc-canary-rollback-predictor-8589bf8f95-9tf7s\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:11.991858 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:11.991736    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5113f39-2584-4bd0-a0e2-fa509bb104d7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-8589bf8f95-9tf7s\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:11.991858 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:11.991771    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-8589bf8f95-9tf7s\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:11.991858 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:11.991789    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxng\" (UniqueName: \"kubernetes.io/projected/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kube-api-access-frxng\") pod \"isvc-canary-rollback-predictor-8589bf8f95-9tf7s\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:11.991858 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:11.991809    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-8589bf8f95-9tf7s\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:12.092789 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.092751    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5113f39-2584-4bd0-a0e2-fa509bb104d7-proxy-tls\") pod \"isvc-canary-rollback-predictor-8589bf8f95-9tf7s\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:12.092789 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.092791    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5113f39-2584-4bd0-a0e2-fa509bb104d7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-8589bf8f95-9tf7s\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:12.093074 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.092841    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-8589bf8f95-9tf7s\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:12.093074 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.092864    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxng\" (UniqueName: \"kubernetes.io/projected/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kube-api-access-frxng\") pod \"isvc-canary-rollback-predictor-8589bf8f95-9tf7s\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:12.093074 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.092888    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-8589bf8f95-9tf7s\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:12.093347 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.093322    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-8589bf8f95-9tf7s\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:12.093606 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.093590    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-8589bf8f95-9tf7s\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:12.093654 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.093599    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5113f39-2584-4bd0-a0e2-fa509bb104d7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-8589bf8f95-9tf7s\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:12.095500 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.095476    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5113f39-2584-4bd0-a0e2-fa509bb104d7-proxy-tls\") pod \"isvc-canary-rollback-predictor-8589bf8f95-9tf7s\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:12.101339 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.101312    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxng\" (UniqueName: \"kubernetes.io/projected/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kube-api-access-frxng\") pod \"isvc-canary-rollback-predictor-8589bf8f95-9tf7s\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:12.285615 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.285517    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:12.433069 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.433038    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"]
Aug 01 00:21:12.436096 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:21:12.436059    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5113f39_2584_4bd0_a0e2_fa509bb104d7.slice/crio-234326fe295abc627f2c10ec10ffdf9c0b0201fdd0e103d2eab08e766ef7caf6 WatchSource:0}: Error finding container 234326fe295abc627f2c10ec10ffdf9c0b0201fdd0e103d2eab08e766ef7caf6: Status 404 returned error can't find the container with id 234326fe295abc627f2c10ec10ffdf9c0b0201fdd0e103d2eab08e766ef7caf6
Aug 01 00:21:12.445580 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.445548    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Aug 01 00:21:12.595973 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.595923    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" event={"ID":"f5113f39-2584-4bd0-a0e2-fa509bb104d7","Type":"ContainerStarted","Data":"536178f470e5c50b9b993f9ae9365e7598a7c09e660ab45c2961004efa52cd3c"}
Aug 01 00:21:12.596124 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.596027    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" event={"ID":"f5113f39-2584-4bd0-a0e2-fa509bb104d7","Type":"ContainerStarted","Data":"234326fe295abc627f2c10ec10ffdf9c0b0201fdd0e103d2eab08e766ef7caf6"}
Aug 01 00:21:12.636783 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.636745    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"]
Aug 01 00:21:12.639402 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.639381    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.641841 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.641817    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Aug 01 00:21:12.641841 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.641835    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Aug 01 00:21:12.650344 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.650323    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"]
Aug 01 00:21:12.697030 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.696988    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-72jm9\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.697204 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.697131    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdlkv\" (UniqueName: \"kubernetes.io/projected/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kube-api-access-bdlkv\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-72jm9\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.697204 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.697195    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-72jm9\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.697339 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.697293    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-72jm9\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.697420 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.697384    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-72jm9\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.797829 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.797798    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-72jm9\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.798030 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.797846    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-72jm9\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.798030 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.797880    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-72jm9\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.798030 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.798010    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdlkv\" (UniqueName: \"kubernetes.io/projected/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kube-api-access-bdlkv\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-72jm9\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.798207 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.798058    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-72jm9\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.798352 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.798325    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-72jm9\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.798552 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.798518    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-72jm9\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.798709 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.798692    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-72jm9\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.800331 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.800314    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-72jm9\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.806039 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.806021    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdlkv\" (UniqueName: \"kubernetes.io/projected/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kube-api-access-bdlkv\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-72jm9\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:12.950867 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:12.950780    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:13.089947 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:13.089923    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"]
Aug 01 00:21:13.092100 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:21:13.092075    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd26a6b3_0e0b_4df3_8015_8a430fa48f38.slice/crio-ccd6c974e5e582bd0148fcb871ce43efaa91630b6c75a2c5be5649a43a2e6887 WatchSource:0}: Error finding container ccd6c974e5e582bd0148fcb871ce43efaa91630b6c75a2c5be5649a43a2e6887: Status 404 returned error can't find the container with id ccd6c974e5e582bd0148fcb871ce43efaa91630b6c75a2c5be5649a43a2e6887
Aug 01 00:21:13.602660 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:13.602622    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" event={"ID":"dd26a6b3-0e0b-4df3-8015-8a430fa48f38","Type":"ContainerStarted","Data":"6884feb1f6a35f341ce2b6f759d393956e4f7e54443ff51713420bc57cb64aea"}
Aug 01 00:21:13.602824 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:13.602689    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" event={"ID":"dd26a6b3-0e0b-4df3-8015-8a430fa48f38","Type":"ContainerStarted","Data":"ccd6c974e5e582bd0148fcb871ce43efaa91630b6c75a2c5be5649a43a2e6887"}
Aug 01 00:21:16.616922 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:16.616891    2584 generic.go:356] "Generic (PLEG): container finished" podID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerID="536178f470e5c50b9b993f9ae9365e7598a7c09e660ab45c2961004efa52cd3c" exitCode=0
Aug 01 00:21:16.617325 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:16.616960    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" event={"ID":"f5113f39-2584-4bd0-a0e2-fa509bb104d7","Type":"ContainerDied","Data":"536178f470e5c50b9b993f9ae9365e7598a7c09e660ab45c2961004efa52cd3c"}
Aug 01 00:21:17.445603 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:17.445565    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Aug 01 00:21:17.445728 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:17.445677    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:21:17.623543 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:17.623497    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" event={"ID":"f5113f39-2584-4bd0-a0e2-fa509bb104d7","Type":"ContainerStarted","Data":"396f5a31b291d13166b079f3ae1499dd735b5253284acbceb9e3537fe92e5fbf"}
Aug 01 00:21:17.624008 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:17.623576    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" event={"ID":"f5113f39-2584-4bd0-a0e2-fa509bb104d7","Type":"ContainerStarted","Data":"fa4d1b17366a7e80ba1638bfe612bfc206014648ad87f655f10cd833044f90b3"}
Aug 01 00:21:17.624008 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:17.623919    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:17.624152 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:17.624072    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:17.625243 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:17.625218    2584 generic.go:356] "Generic (PLEG): container finished" podID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerID="6884feb1f6a35f341ce2b6f759d393956e4f7e54443ff51713420bc57cb64aea" exitCode=0
Aug 01 00:21:17.625348 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:17.625275    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" event={"ID":"dd26a6b3-0e0b-4df3-8015-8a430fa48f38","Type":"ContainerDied","Data":"6884feb1f6a35f341ce2b6f759d393956e4f7e54443ff51713420bc57cb64aea"}
Aug 01 00:21:17.625706 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:17.625681    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:21:17.643332 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:17.643281    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" podStartSLOduration=3.001141615 podStartE2EDuration="6.643264224s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.642122609s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:21:17.641254707 +0000 UTC m=+2112.514898323" watchObservedRunningTime="2026-08-01 00:21:17.643264224 +0000 UTC m=+2112.516907837"
Aug 01 00:21:18.631474 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:18.631439    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" event={"ID":"dd26a6b3-0e0b-4df3-8015-8a430fa48f38","Type":"ContainerStarted","Data":"b670f16d9b3930a91a164be115c82c92491a30a467ec668e3078da8ed8862bb4"}
Aug 01 00:21:18.631886 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:18.631512    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" event={"ID":"dd26a6b3-0e0b-4df3-8015-8a430fa48f38","Type":"ContainerStarted","Data":"a228aadfb3902a523377a973091ba11cb5eb98a8965c1929f1ceeabc9ecb7046"}
Aug 01 00:21:18.631886 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:18.631682    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:21:18.631886 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:18.631735    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:19.635731 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:19.635697    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:19.636919 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:19.636897    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 01 00:21:20.640206 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:20.640151    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 01 00:21:22.445106 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:22.445063    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Aug 01 00:21:23.636015 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:23.635987    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:21:23.636498 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:23.636472    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:21:23.656619 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:23.656573    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" podStartSLOduration=7.385785402 podStartE2EDuration="11.656557917s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.270772515s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:21:18.650701501 +0000 UTC m=+2113.524345108" watchObservedRunningTime="2026-08-01 00:21:23.656557917 +0000 UTC m=+2118.530201528"
Aug 01 00:21:25.645503 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:25.645472    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:25.646281 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:25.646255    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 01 00:21:27.444691 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:27.444643    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Aug 01 00:21:32.445168 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:32.445123    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Aug 01 00:21:33.636545 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:33.636503    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:21:35.646478 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:35.646447    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:21:37.444919 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:37.444879    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Aug 01 00:21:37.713000 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:37.712896    2584 generic.go:356] "Generic (PLEG): container finished" podID="aee6e647-f05f-4903-8405-94a54f62e273" containerID="c133d0da510e340496db947c7468c46b01a15fcb2599b04490c68976a2f89dfd" exitCode=137
Aug 01 00:21:37.713000 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:37.712989    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" event={"ID":"aee6e647-f05f-4903-8405-94a54f62e273","Type":"ContainerDied","Data":"c133d0da510e340496db947c7468c46b01a15fcb2599b04490c68976a2f89dfd"}
Aug 01 00:21:37.992827 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:37.992805    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:21:38.111197 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.111167    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aee6e647-f05f-4903-8405-94a54f62e273-kube-rbac-proxy-sar-config\") pod \"aee6e647-f05f-4903-8405-94a54f62e273\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") "
Aug 01 00:21:38.111197 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.111200    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aee6e647-f05f-4903-8405-94a54f62e273-kserve-provision-location\") pod \"aee6e647-f05f-4903-8405-94a54f62e273\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") "
Aug 01 00:21:38.111400 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.111254    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x64wv\" (UniqueName: \"kubernetes.io/projected/aee6e647-f05f-4903-8405-94a54f62e273-kube-api-access-x64wv\") pod \"aee6e647-f05f-4903-8405-94a54f62e273\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") "
Aug 01 00:21:38.111400 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.111356    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aee6e647-f05f-4903-8405-94a54f62e273-proxy-tls\") pod \"aee6e647-f05f-4903-8405-94a54f62e273\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") "
Aug 01 00:21:38.111485 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.111411    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aee6e647-f05f-4903-8405-94a54f62e273-cabundle-cert\") pod \"aee6e647-f05f-4903-8405-94a54f62e273\" (UID: \"aee6e647-f05f-4903-8405-94a54f62e273\") "
Aug 01 00:21:38.111603 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.111581    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee6e647-f05f-4903-8405-94a54f62e273-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aee6e647-f05f-4903-8405-94a54f62e273" (UID: "aee6e647-f05f-4903-8405-94a54f62e273"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:21:38.111822 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.111801    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aee6e647-f05f-4903-8405-94a54f62e273-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:21:38.111822 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.111807    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee6e647-f05f-4903-8405-94a54f62e273-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aee6e647-f05f-4903-8405-94a54f62e273" (UID: "aee6e647-f05f-4903-8405-94a54f62e273"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:21:38.113281 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.113263    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee6e647-f05f-4903-8405-94a54f62e273-kube-api-access-x64wv" (OuterVolumeSpecName: "kube-api-access-x64wv") pod "aee6e647-f05f-4903-8405-94a54f62e273" (UID: "aee6e647-f05f-4903-8405-94a54f62e273"). InnerVolumeSpecName "kube-api-access-x64wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:21:38.113400 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.113387    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee6e647-f05f-4903-8405-94a54f62e273-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aee6e647-f05f-4903-8405-94a54f62e273" (UID: "aee6e647-f05f-4903-8405-94a54f62e273"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:21:38.122827 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.122797    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee6e647-f05f-4903-8405-94a54f62e273-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aee6e647-f05f-4903-8405-94a54f62e273" (UID: "aee6e647-f05f-4903-8405-94a54f62e273"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:21:38.212623 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.212588    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aee6e647-f05f-4903-8405-94a54f62e273-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:21:38.212623 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.212618    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x64wv\" (UniqueName: \"kubernetes.io/projected/aee6e647-f05f-4903-8405-94a54f62e273-kube-api-access-x64wv\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:21:38.212623 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.212630    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aee6e647-f05f-4903-8405-94a54f62e273-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:21:38.212851 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.212644    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aee6e647-f05f-4903-8405-94a54f62e273-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:21:38.719481 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.719451    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"
Aug 01 00:21:38.719959 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.719447    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g" event={"ID":"aee6e647-f05f-4903-8405-94a54f62e273","Type":"ContainerDied","Data":"bf63959ab96312b3a57c4ffbe5e734e3218f9345b685795f46a6768a375f79e2"}
Aug 01 00:21:38.719959 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.719590    2584 scope.go:117] "RemoveContainer" containerID="3ce377673568ba2c0d3997c61b08e763c708f813a27cd6ddc293687c9525e5c4"
Aug 01 00:21:38.728541 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.728524    2584 scope.go:117] "RemoveContainer" containerID="c133d0da510e340496db947c7468c46b01a15fcb2599b04490c68976a2f89dfd"
Aug 01 00:21:38.735681 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.735662    2584 scope.go:117] "RemoveContainer" containerID="8e5010bf3a0f9953c046955dfb64eb4bd236f094bcb5f2e3d4a635a72d32d842"
Aug 01 00:21:38.740994 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.740952    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"]
Aug 01 00:21:38.744260 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:38.744241    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5l76g"]
Aug 01 00:21:39.570460 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:39.570429    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aee6e647-f05f-4903-8405-94a54f62e273" path="/var/lib/kubelet/pods/aee6e647-f05f-4903-8405-94a54f62e273/volumes"
Aug 01 00:21:43.636640 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:43.636607    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:21:53.636991 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:53.636931    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:21:53.651668 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:53.651636    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"]
Aug 01 00:21:53.651901 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:53.651871    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kserve-container" containerID="cri-o://a228aadfb3902a523377a973091ba11cb5eb98a8965c1929f1ceeabc9ecb7046" gracePeriod=30
Aug 01 00:21:53.652053 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:53.651894    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kube-rbac-proxy" containerID="cri-o://b670f16d9b3930a91a164be115c82c92491a30a467ec668e3078da8ed8862bb4" gracePeriod=30
Aug 01 00:21:53.787739 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:53.787704    2584 generic.go:356] "Generic (PLEG): container finished" podID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerID="b670f16d9b3930a91a164be115c82c92491a30a467ec668e3078da8ed8862bb4" exitCode=2
Aug 01 00:21:53.787904 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:53.787777    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" event={"ID":"dd26a6b3-0e0b-4df3-8015-8a430fa48f38","Type":"ContainerDied","Data":"b670f16d9b3930a91a164be115c82c92491a30a467ec668e3078da8ed8862bb4"}
Aug 01 00:21:55.640779 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:21:55.640736    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Aug 01 00:22:00.640713 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:00.640620    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Aug 01 00:22:03.637431 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:03.637391    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:22:05.640305 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:05.640262    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Aug 01 00:22:05.640684 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:05.640383    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:22:10.641062 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:10.641021    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Aug 01 00:22:13.636657 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:13.636619    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:22:15.640587 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:15.640544    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Aug 01 00:22:20.640398 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:20.640352    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Aug 01 00:22:23.636517 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:23.636475    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:22:23.930106 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:23.930014    2584 generic.go:356] "Generic (PLEG): container finished" podID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerID="a228aadfb3902a523377a973091ba11cb5eb98a8965c1929f1ceeabc9ecb7046" exitCode=137
Aug 01 00:22:23.930304 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:23.930096    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" event={"ID":"dd26a6b3-0e0b-4df3-8015-8a430fa48f38","Type":"ContainerDied","Data":"a228aadfb3902a523377a973091ba11cb5eb98a8965c1929f1ceeabc9ecb7046"}
Aug 01 00:22:24.300242 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.300210    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:22:24.403884 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.403841    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-proxy-tls\") pod \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") "
Aug 01 00:22:24.404090 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.403898    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdlkv\" (UniqueName: \"kubernetes.io/projected/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kube-api-access-bdlkv\") pod \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") "
Aug 01 00:22:24.404090 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.403925    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kserve-provision-location\") pod \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") "
Aug 01 00:22:24.404090 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.403956    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-cabundle-cert\") pod \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") "
Aug 01 00:22:24.404090 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.404007    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kube-rbac-proxy-sar-config\") pod \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\" (UID: \"dd26a6b3-0e0b-4df3-8015-8a430fa48f38\") "
Aug 01 00:22:24.404372 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.404341    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd26a6b3-0e0b-4df3-8015-8a430fa48f38" (UID: "dd26a6b3-0e0b-4df3-8015-8a430fa48f38"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:22:24.404486 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.404445    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd26a6b3-0e0b-4df3-8015-8a430fa48f38" (UID: "dd26a6b3-0e0b-4df3-8015-8a430fa48f38"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:22:24.406191 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.406159    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd26a6b3-0e0b-4df3-8015-8a430fa48f38" (UID: "dd26a6b3-0e0b-4df3-8015-8a430fa48f38"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:22:24.406283 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.406267    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kube-api-access-bdlkv" (OuterVolumeSpecName: "kube-api-access-bdlkv") pod "dd26a6b3-0e0b-4df3-8015-8a430fa48f38" (UID: "dd26a6b3-0e0b-4df3-8015-8a430fa48f38"). InnerVolumeSpecName "kube-api-access-bdlkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:22:24.413160 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.413133    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd26a6b3-0e0b-4df3-8015-8a430fa48f38" (UID: "dd26a6b3-0e0b-4df3-8015-8a430fa48f38"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:22:24.505245 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.505156    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdlkv\" (UniqueName: \"kubernetes.io/projected/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kube-api-access-bdlkv\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:22:24.505245 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.505186    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:22:24.505245 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.505196    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:22:24.505245 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.505205    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:22:24.505245 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.505216    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd26a6b3-0e0b-4df3-8015-8a430fa48f38-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:22:24.936142 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.936113    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"
Aug 01 00:22:24.936623 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.936117    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9" event={"ID":"dd26a6b3-0e0b-4df3-8015-8a430fa48f38","Type":"ContainerDied","Data":"ccd6c974e5e582bd0148fcb871ce43efaa91630b6c75a2c5be5649a43a2e6887"}
Aug 01 00:22:24.936623 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.936258    2584 scope.go:117] "RemoveContainer" containerID="b670f16d9b3930a91a164be115c82c92491a30a467ec668e3078da8ed8862bb4"
Aug 01 00:22:24.944879 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.944484    2584 scope.go:117] "RemoveContainer" containerID="a228aadfb3902a523377a973091ba11cb5eb98a8965c1929f1ceeabc9ecb7046"
Aug 01 00:22:24.952469 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.952449    2584 scope.go:117] "RemoveContainer" containerID="6884feb1f6a35f341ce2b6f759d393956e4f7e54443ff51713420bc57cb64aea"
Aug 01 00:22:24.957756 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.957733    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"]
Aug 01 00:22:24.961940 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:24.961920    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-72jm9"]
Aug 01 00:22:25.570390 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:25.570355    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" path="/var/lib/kubelet/pods/dd26a6b3-0e0b-4df3-8015-8a430fa48f38/volumes"
Aug 01 00:22:33.637684 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:33.637651    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:22:42.070838 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:42.070809    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"]
Aug 01 00:22:42.071432 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:42.071096    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kserve-container" containerID="cri-o://fa4d1b17366a7e80ba1638bfe612bfc206014648ad87f655f10cd833044f90b3" gracePeriod=30
Aug 01 00:22:42.071432 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:42.071134    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kube-rbac-proxy" containerID="cri-o://396f5a31b291d13166b079f3ae1499dd735b5253284acbceb9e3537fe92e5fbf" gracePeriod=30
Aug 01 00:22:43.015693 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:43.015651    2584 generic.go:356] "Generic (PLEG): container finished" podID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerID="396f5a31b291d13166b079f3ae1499dd735b5253284acbceb9e3537fe92e5fbf" exitCode=2
Aug 01 00:22:43.015882 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:43.015727    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" event={"ID":"f5113f39-2584-4bd0-a0e2-fa509bb104d7","Type":"ContainerDied","Data":"396f5a31b291d13166b079f3ae1499dd735b5253284acbceb9e3537fe92e5fbf"}
Aug 01 00:22:43.631921 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:43.631877    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Aug 01 00:22:43.637095 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:43.637072    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:22:46.723089 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.723065    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:22:46.790356 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.790329    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frxng\" (UniqueName: \"kubernetes.io/projected/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kube-api-access-frxng\") pod \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") "
Aug 01 00:22:46.790547 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.790402    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kserve-provision-location\") pod \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") "
Aug 01 00:22:46.790547 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.790434    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kube-rbac-proxy-sar-config\") pod \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") "
Aug 01 00:22:46.790547 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.790477    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5113f39-2584-4bd0-a0e2-fa509bb104d7-proxy-tls\") pod \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") "
Aug 01 00:22:46.790547 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.790512    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5113f39-2584-4bd0-a0e2-fa509bb104d7-cabundle-cert\") pod \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\" (UID: \"f5113f39-2584-4bd0-a0e2-fa509bb104d7\") "
Aug 01 00:22:46.790793 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.790764    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5113f39-2584-4bd0-a0e2-fa509bb104d7" (UID: "f5113f39-2584-4bd0-a0e2-fa509bb104d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:22:46.790845 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.790803    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5113f39-2584-4bd0-a0e2-fa509bb104d7" (UID: "f5113f39-2584-4bd0-a0e2-fa509bb104d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:22:46.790951 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.790920    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5113f39-2584-4bd0-a0e2-fa509bb104d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5113f39-2584-4bd0-a0e2-fa509bb104d7" (UID: "f5113f39-2584-4bd0-a0e2-fa509bb104d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:22:46.792424 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.792402    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kube-api-access-frxng" (OuterVolumeSpecName: "kube-api-access-frxng") pod "f5113f39-2584-4bd0-a0e2-fa509bb104d7" (UID: "f5113f39-2584-4bd0-a0e2-fa509bb104d7"). InnerVolumeSpecName "kube-api-access-frxng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:22:46.792523 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.792483    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5113f39-2584-4bd0-a0e2-fa509bb104d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5113f39-2584-4bd0-a0e2-fa509bb104d7" (UID: "f5113f39-2584-4bd0-a0e2-fa509bb104d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:22:46.891985 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.891923    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frxng\" (UniqueName: \"kubernetes.io/projected/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kube-api-access-frxng\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:22:46.891985 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.891955    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:22:46.891985 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.891989    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5113f39-2584-4bd0-a0e2-fa509bb104d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:22:46.892232 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.892001    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5113f39-2584-4bd0-a0e2-fa509bb104d7-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:22:46.892232 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:46.892010    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5113f39-2584-4bd0-a0e2-fa509bb104d7-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:22:47.038644 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.038608    2584 generic.go:356] "Generic (PLEG): container finished" podID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerID="fa4d1b17366a7e80ba1638bfe612bfc206014648ad87f655f10cd833044f90b3" exitCode=0
Aug 01 00:22:47.038644 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.038646    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" event={"ID":"f5113f39-2584-4bd0-a0e2-fa509bb104d7","Type":"ContainerDied","Data":"fa4d1b17366a7e80ba1638bfe612bfc206014648ad87f655f10cd833044f90b3"}
Aug 01 00:22:47.038865 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.038660    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"
Aug 01 00:22:47.038865 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.038694    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s" event={"ID":"f5113f39-2584-4bd0-a0e2-fa509bb104d7","Type":"ContainerDied","Data":"234326fe295abc627f2c10ec10ffdf9c0b0201fdd0e103d2eab08e766ef7caf6"}
Aug 01 00:22:47.038865 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.038721    2584 scope.go:117] "RemoveContainer" containerID="396f5a31b291d13166b079f3ae1499dd735b5253284acbceb9e3537fe92e5fbf"
Aug 01 00:22:47.047255 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.047236    2584 scope.go:117] "RemoveContainer" containerID="fa4d1b17366a7e80ba1638bfe612bfc206014648ad87f655f10cd833044f90b3"
Aug 01 00:22:47.054406 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.054390    2584 scope.go:117] "RemoveContainer" containerID="536178f470e5c50b9b993f9ae9365e7598a7c09e660ab45c2961004efa52cd3c"
Aug 01 00:22:47.061532 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.061509    2584 scope.go:117] "RemoveContainer" containerID="396f5a31b291d13166b079f3ae1499dd735b5253284acbceb9e3537fe92e5fbf"
Aug 01 00:22:47.061869 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:22:47.061841    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396f5a31b291d13166b079f3ae1499dd735b5253284acbceb9e3537fe92e5fbf\": container with ID starting with 396f5a31b291d13166b079f3ae1499dd735b5253284acbceb9e3537fe92e5fbf not found: ID does not exist" containerID="396f5a31b291d13166b079f3ae1499dd735b5253284acbceb9e3537fe92e5fbf"
Aug 01 00:22:47.061957 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.061881    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396f5a31b291d13166b079f3ae1499dd735b5253284acbceb9e3537fe92e5fbf"} err="failed to get container status \"396f5a31b291d13166b079f3ae1499dd735b5253284acbceb9e3537fe92e5fbf\": rpc error: code = NotFound desc = could not find container \"396f5a31b291d13166b079f3ae1499dd735b5253284acbceb9e3537fe92e5fbf\": container with ID starting with 396f5a31b291d13166b079f3ae1499dd735b5253284acbceb9e3537fe92e5fbf not found: ID does not exist"
Aug 01 00:22:47.061957 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.061902    2584 scope.go:117] "RemoveContainer" containerID="fa4d1b17366a7e80ba1638bfe612bfc206014648ad87f655f10cd833044f90b3"
Aug 01 00:22:47.062277 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.062260    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"]
Aug 01 00:22:47.062374 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:22:47.062276    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa4d1b17366a7e80ba1638bfe612bfc206014648ad87f655f10cd833044f90b3\": container with ID starting with fa4d1b17366a7e80ba1638bfe612bfc206014648ad87f655f10cd833044f90b3 not found: ID does not exist" containerID="fa4d1b17366a7e80ba1638bfe612bfc206014648ad87f655f10cd833044f90b3"
Aug 01 00:22:47.062374 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.062301    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa4d1b17366a7e80ba1638bfe612bfc206014648ad87f655f10cd833044f90b3"} err="failed to get container status \"fa4d1b17366a7e80ba1638bfe612bfc206014648ad87f655f10cd833044f90b3\": rpc error: code = NotFound desc = could not find container \"fa4d1b17366a7e80ba1638bfe612bfc206014648ad87f655f10cd833044f90b3\": container with ID starting with fa4d1b17366a7e80ba1638bfe612bfc206014648ad87f655f10cd833044f90b3 not found: ID does not exist"
Aug 01 00:22:47.062374 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.062321    2584 scope.go:117] "RemoveContainer" containerID="536178f470e5c50b9b993f9ae9365e7598a7c09e660ab45c2961004efa52cd3c"
Aug 01 00:22:47.062573 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:22:47.062555    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"536178f470e5c50b9b993f9ae9365e7598a7c09e660ab45c2961004efa52cd3c\": container with ID starting with 536178f470e5c50b9b993f9ae9365e7598a7c09e660ab45c2961004efa52cd3c not found: ID does not exist" containerID="536178f470e5c50b9b993f9ae9365e7598a7c09e660ab45c2961004efa52cd3c"
Aug 01 00:22:47.062613 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.062580    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"536178f470e5c50b9b993f9ae9365e7598a7c09e660ab45c2961004efa52cd3c"} err="failed to get container status \"536178f470e5c50b9b993f9ae9365e7598a7c09e660ab45c2961004efa52cd3c\": rpc error: code = NotFound desc = could not find container \"536178f470e5c50b9b993f9ae9365e7598a7c09e660ab45c2961004efa52cd3c\": container with ID starting with 536178f470e5c50b9b993f9ae9365e7598a7c09e660ab45c2961004efa52cd3c not found: ID does not exist"
Aug 01 00:22:47.063702 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.063683    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-9tf7s"]
Aug 01 00:22:47.274338 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274308    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"]
Aug 01 00:22:47.274619 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274606    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kserve-container"
Aug 01 00:22:47.274662 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274621    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kserve-container"
Aug 01 00:22:47.274662 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274635    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="storage-initializer"
Aug 01 00:22:47.274662 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274641    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="storage-initializer"
Aug 01 00:22:47.274662 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274649    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="storage-initializer"
Aug 01 00:22:47.274662 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274656    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="storage-initializer"
Aug 01 00:22:47.274662 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274663    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="storage-initializer"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274669    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="storage-initializer"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274676    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kserve-container"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274681    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kserve-container"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274687    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kube-rbac-proxy"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274692    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kube-rbac-proxy"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274698    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kube-rbac-proxy"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274704    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kube-rbac-proxy"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274709    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kserve-container"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274714    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kserve-container"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274719    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kube-rbac-proxy"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274724    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kube-rbac-proxy"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274777    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kube-rbac-proxy"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274785    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kube-rbac-proxy"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274791    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee6e647-f05f-4903-8405-94a54f62e273" containerName="kserve-container"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274797    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kube-rbac-proxy"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274803    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" containerName="kserve-container"
Aug 01 00:22:47.274844 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.274810    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd26a6b3-0e0b-4df3-8015-8a430fa48f38" containerName="kserve-container"
Aug 01 00:22:47.279220 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.279203    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.281824 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.281793    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:22:47.281983 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.281831    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Aug 01 00:22:47.281983 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.281857    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:22:47.281983 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.281886    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Aug 01 00:22:47.281983 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.281831    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:22:47.282250 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.282237    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:22:47.282303 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.282257    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:22:47.288659 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.288636    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"]
Aug 01 00:22:47.314335 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.314271    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"]
Aug 01 00:22:47.318996 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.318944    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:47.321869 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.321847    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Aug 01 00:22:47.329263 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.329241    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"]
Aug 01 00:22:47.395600 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.395568    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g46p\" (UniqueName: \"kubernetes.io/projected/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kube-api-access-2g46p\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:47.395776 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.395627    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:47.395776 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.395659    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:47.395776 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.395691    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l4c7\" (UniqueName: \"kubernetes.io/projected/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kube-api-access-2l4c7\") pod \"isvc-canary-stop-predictor-697f894f79-gmqgc\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.395776 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.395719    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-cabundle-cert\") pod \"isvc-canary-stop-predictor-697f894f79-gmqgc\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.396029 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.395805    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-proxy-tls\") pod \"isvc-canary-stop-predictor-697f894f79-gmqgc\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.396029 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.395853    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:47.396029 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.395882    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-697f894f79-gmqgc\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.396029 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.395898    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kserve-provision-location\") pod \"isvc-canary-stop-predictor-697f894f79-gmqgc\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.396029 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.395919    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:47.496675 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.496639    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:47.496822 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.496685    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:47.496822 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.496717    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4c7\" (UniqueName: \"kubernetes.io/projected/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kube-api-access-2l4c7\") pod \"isvc-canary-stop-predictor-697f894f79-gmqgc\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.496822 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.496742    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-cabundle-cert\") pod \"isvc-canary-stop-predictor-697f894f79-gmqgc\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.496822 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.496805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-proxy-tls\") pod \"isvc-canary-stop-predictor-697f894f79-gmqgc\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.497067 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.496850    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:47.497067 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.496892    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-697f894f79-gmqgc\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.497067 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.497046    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kserve-provision-location\") pod \"isvc-canary-stop-predictor-697f894f79-gmqgc\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.497222 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.497093    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:47.497222 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.497122    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:47.497222 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.497136    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g46p\" (UniqueName: \"kubernetes.io/projected/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kube-api-access-2g46p\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:47.497450 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:22:47.497432    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Aug 01 00:22:47.497507 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.497445    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:47.497507 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.497469    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kserve-provision-location\") pod \"isvc-canary-stop-predictor-697f894f79-gmqgc\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.497616 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:22:47.497528    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-proxy-tls podName:69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff nodeName:}" failed. No retries permitted until 2026-08-01 00:22:47.997507653 +0000 UTC m=+2202.871151265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-proxy-tls") pod "isvc-canary-stop-v2-predictor-6688bb854b-knppv" (UID: "69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Aug 01 00:22:47.497616 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.497553    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-697f894f79-gmqgc\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.497616 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.497610    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:47.497799 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.497779    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-cabundle-cert\") pod \"isvc-canary-stop-predictor-697f894f79-gmqgc\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.499417 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.499395    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-proxy-tls\") pod \"isvc-canary-stop-predictor-697f894f79-gmqgc\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.505660 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.505642    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g46p\" (UniqueName: \"kubernetes.io/projected/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kube-api-access-2g46p\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:47.505842 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.505825    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l4c7\" (UniqueName: \"kubernetes.io/projected/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kube-api-access-2l4c7\") pod \"isvc-canary-stop-predictor-697f894f79-gmqgc\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.571264 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.571193    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5113f39-2584-4bd0-a0e2-fa509bb104d7" path="/var/lib/kubelet/pods/f5113f39-2584-4bd0-a0e2-fa509bb104d7/volumes"
Aug 01 00:22:47.590760 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.590732    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:47.719815 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:47.719783    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"]
Aug 01 00:22:47.722868 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:22:47.722843    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c5df2c_d17b_49cb_8ef8_e1fbff732bfa.slice/crio-c790ffb13c95e150aae1f2ec52547c1e8709ec060565128567ea2ffda686a389 WatchSource:0}: Error finding container c790ffb13c95e150aae1f2ec52547c1e8709ec060565128567ea2ffda686a389: Status 404 returned error can't find the container with id c790ffb13c95e150aae1f2ec52547c1e8709ec060565128567ea2ffda686a389
Aug 01 00:22:48.000762 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:48.000684    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:48.003318 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:48.003293    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-knppv\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:48.045296 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:48.045261    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" event={"ID":"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa","Type":"ContainerStarted","Data":"a1ac3610c4d776e255cde4b4babf6f6508d5193c350a38bc029be0c90c8be835"}
Aug 01 00:22:48.045473 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:48.045335    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" event={"ID":"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa","Type":"ContainerStarted","Data":"c790ffb13c95e150aae1f2ec52547c1e8709ec060565128567ea2ffda686a389"}
Aug 01 00:22:48.230919 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:48.230884    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:48.358578 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:48.358550    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"]
Aug 01 00:22:48.360176 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:22:48.360152    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d8486c_0aa6_4f10_b5f0_e6efbe3a6eff.slice/crio-3af73639b90a4425d4eb2bee253525d773b345d2f593ade14f44fe2c2820fa60 WatchSource:0}: Error finding container 3af73639b90a4425d4eb2bee253525d773b345d2f593ade14f44fe2c2820fa60: Status 404 returned error can't find the container with id 3af73639b90a4425d4eb2bee253525d773b345d2f593ade14f44fe2c2820fa60
Aug 01 00:22:49.052031 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:49.051991    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" event={"ID":"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff","Type":"ContainerStarted","Data":"acde00a59bcbfcb6c48c26d23b431a63516e44a51570e84027c2c52ed2407ecc"}
Aug 01 00:22:49.052440 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:49.052062    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" event={"ID":"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff","Type":"ContainerStarted","Data":"3af73639b90a4425d4eb2bee253525d773b345d2f593ade14f44fe2c2820fa60"}
Aug 01 00:22:52.066214 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:52.066179    2584 generic.go:356] "Generic (PLEG): container finished" podID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerID="a1ac3610c4d776e255cde4b4babf6f6508d5193c350a38bc029be0c90c8be835" exitCode=0
Aug 01 00:22:52.066675 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:52.066253    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" event={"ID":"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa","Type":"ContainerDied","Data":"a1ac3610c4d776e255cde4b4babf6f6508d5193c350a38bc029be0c90c8be835"}
Aug 01 00:22:53.072206 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:53.072169    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" event={"ID":"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa","Type":"ContainerStarted","Data":"22aa1191e56f80d24db5f1d591bcfec381396f4e4226b0a749a6f57b91eb041d"}
Aug 01 00:22:53.072648 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:53.072244    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" event={"ID":"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa","Type":"ContainerStarted","Data":"5b6ce5a160c7de77ddf8083ce28b518ccf921096ed1bdc9feda0d119bc3f5cbc"}
Aug 01 00:22:53.072648 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:53.072434    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:53.073667 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:53.073645    2584 generic.go:356] "Generic (PLEG): container finished" podID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerID="acde00a59bcbfcb6c48c26d23b431a63516e44a51570e84027c2c52ed2407ecc" exitCode=0
Aug 01 00:22:53.073770 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:53.073720    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" event={"ID":"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff","Type":"ContainerDied","Data":"acde00a59bcbfcb6c48c26d23b431a63516e44a51570e84027c2c52ed2407ecc"}
Aug 01 00:22:53.096088 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:53.096042    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" podStartSLOduration=2.320428273 podStartE2EDuration="6.096020721s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.775592448s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:22:53.095462602 +0000 UTC m=+2207.969106224" watchObservedRunningTime="2026-08-01 00:22:53.096020721 +0000 UTC m=+2207.969664330"
Aug 01 00:22:54.079980 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:54.079932    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" event={"ID":"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff","Type":"ContainerStarted","Data":"d5b37e224620d1e75fd56afabf6c8da34960a6630ab5d81e90505b39bf154565"}
Aug 01 00:22:54.080415 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:54.080020    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" event={"ID":"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff","Type":"ContainerStarted","Data":"72d72338b13db3e819a54561baa553b10b3c095d261672439c391c4421c5a71e"}
Aug 01 00:22:54.080415 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:54.080278    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:22:54.080583 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:54.080566    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:54.080745 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:54.080724    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:22:54.081829 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:54.081808    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:22:54.081926 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:54.081908    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:22:54.100832 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:54.100781    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" podStartSLOduration=3.288767347 podStartE2EDuration="7.100765323s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.811997976s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:22:54.099111907 +0000 UTC m=+2208.972755508" watchObservedRunningTime="2026-08-01 00:22:54.100765323 +0000 UTC m=+2208.974408932"
Aug 01 00:22:55.084980 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:55.084936    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:22:55.085439 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:55.084936    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:22:56.088499 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:22:56.088459    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:23:00.089028 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:23:00.088997    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:23:00.089498 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:23:00.089469    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:23:01.093567 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:23:01.093538    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:23:01.094040 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:23:01.094015    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:23:10.089523 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:23:10.089484    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:23:11.094975 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:23:11.094935    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:23:20.089623 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:23:20.089580    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:23:21.094954 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:23:21.094918    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:23:30.090475 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:23:30.090385    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:23:31.094131 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:23:31.094095    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:23:40.090105 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:23:40.090062    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:23:41.094426 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:23:41.094387    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:23:50.090156 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:23:50.090118    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:23:51.093958 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:23:51.093921    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:24:00.089777 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:00.089741    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:24:01.094210 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:01.094167    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:24:07.571508 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:07.571478    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:24:11.095205 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:11.095168    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:24:17.296865 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:17.296825    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"]
Aug 01 00:24:17.297381 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:17.297156    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kserve-container" containerID="cri-o://5b6ce5a160c7de77ddf8083ce28b518ccf921096ed1bdc9feda0d119bc3f5cbc" gracePeriod=30
Aug 01 00:24:17.297381 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:17.297183    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kube-rbac-proxy" containerID="cri-o://22aa1191e56f80d24db5f1d591bcfec381396f4e4226b0a749a6f57b91eb041d" gracePeriod=30
Aug 01 00:24:17.331110 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:17.331080    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"]
Aug 01 00:24:17.331334 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:17.331311    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kserve-container" containerID="cri-o://72d72338b13db3e819a54561baa553b10b3c095d261672439c391c4421c5a71e" gracePeriod=30
Aug 01 00:24:17.331424 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:17.331370    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kube-rbac-proxy" containerID="cri-o://d5b37e224620d1e75fd56afabf6c8da34960a6630ab5d81e90505b39bf154565" gracePeriod=30
Aug 01 00:24:17.456224 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:17.456191    2584 generic.go:356] "Generic (PLEG): container finished" podID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerID="22aa1191e56f80d24db5f1d591bcfec381396f4e4226b0a749a6f57b91eb041d" exitCode=2
Aug 01 00:24:17.456392 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:17.456269    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" event={"ID":"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa","Type":"ContainerDied","Data":"22aa1191e56f80d24db5f1d591bcfec381396f4e4226b0a749a6f57b91eb041d"}
Aug 01 00:24:17.458362 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:17.458339    2584 generic.go:356] "Generic (PLEG): container finished" podID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerID="d5b37e224620d1e75fd56afabf6c8da34960a6630ab5d81e90505b39bf154565" exitCode=2
Aug 01 00:24:17.458713 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:17.458389    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" event={"ID":"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff","Type":"ContainerDied","Data":"d5b37e224620d1e75fd56afabf6c8da34960a6630ab5d81e90505b39bf154565"}
Aug 01 00:24:17.566618 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:17.566533    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:24:20.085271 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:20.085225    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Aug 01 00:24:21.089239 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:21.089197    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Aug 01 00:24:21.094051 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:21.094017    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:24:22.490925 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.490892    2584 generic.go:356] "Generic (PLEG): container finished" podID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerID="5b6ce5a160c7de77ddf8083ce28b518ccf921096ed1bdc9feda0d119bc3f5cbc" exitCode=0
Aug 01 00:24:22.491357 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.490977    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" event={"ID":"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa","Type":"ContainerDied","Data":"5b6ce5a160c7de77ddf8083ce28b518ccf921096ed1bdc9feda0d119bc3f5cbc"}
Aug 01 00:24:22.493887 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.493858    2584 generic.go:356] "Generic (PLEG): container finished" podID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerID="72d72338b13db3e819a54561baa553b10b3c095d261672439c391c4421c5a71e" exitCode=0
Aug 01 00:24:22.494067 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.493947    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" event={"ID":"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff","Type":"ContainerDied","Data":"72d72338b13db3e819a54561baa553b10b3c095d261672439c391c4421c5a71e"}
Aug 01 00:24:22.566828 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.566792    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:24:22.569259 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.569243    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:24:22.687204 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.687116    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-proxy-tls\") pod \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") "
Aug 01 00:24:22.687204 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.687160    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kserve-provision-location\") pod \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") "
Aug 01 00:24:22.687204 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.687184    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l4c7\" (UniqueName: \"kubernetes.io/projected/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kube-api-access-2l4c7\") pod \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") "
Aug 01 00:24:22.687204 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.687205    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-cabundle-cert\") pod \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") "
Aug 01 00:24:22.687540 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.687318    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kube-rbac-proxy-sar-config\") pod \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") "
Aug 01 00:24:22.687540 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.687370    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g46p\" (UniqueName: \"kubernetes.io/projected/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kube-api-access-2g46p\") pod \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") "
Aug 01 00:24:22.687540 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.687434    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-proxy-tls\") pod \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") "
Aug 01 00:24:22.687540 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.687474    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-cabundle-cert\") pod \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") "
Aug 01 00:24:22.687540 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.687526    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kube-rbac-proxy-sar-config\") pod \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\" (UID: \"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa\") "
Aug 01 00:24:22.687806 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.687553    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kserve-provision-location\") pod \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\" (UID: \"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff\") "
Aug 01 00:24:22.687806 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.687569    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" (UID: "f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:24:22.687806 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.687579    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" (UID: "f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:22.687806 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.687682    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" (UID: "69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:24:22.688076 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.687950    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" (UID: "69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:24:22.688076 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.687957    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:24:22.688076 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.688003    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:24:22.688076 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.688009    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" (UID: "69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:22.688076 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.688018    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:24:22.688342 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.688224    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" (UID: "f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:24:22.689597 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.689572    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" (UID: "f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:24:22.689597 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.689582    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kube-api-access-2l4c7" (OuterVolumeSpecName: "kube-api-access-2l4c7") pod "f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" (UID: "f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa"). InnerVolumeSpecName "kube-api-access-2l4c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:24:22.689749 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.689577    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" (UID: "69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:24:22.689749 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.689715    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kube-api-access-2g46p" (OuterVolumeSpecName: "kube-api-access-2g46p") pod "69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" (UID: "69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff"). InnerVolumeSpecName "kube-api-access-2g46p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:24:22.789133 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.789102    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:24:22.789133 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.789128    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l4c7\" (UniqueName: \"kubernetes.io/projected/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kube-api-access-2l4c7\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:24:22.789133 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.789139    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g46p\" (UniqueName: \"kubernetes.io/projected/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kube-api-access-2g46p\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:24:22.789348 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.789148    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:24:22.789348 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.789157    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:24:22.789348 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.789165    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:24:22.789348 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:22.789175    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:24:23.500206 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.500165    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc" event={"ID":"f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa","Type":"ContainerDied","Data":"c790ffb13c95e150aae1f2ec52547c1e8709ec060565128567ea2ffda686a389"}
Aug 01 00:24:23.500666 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.500244    2584 scope.go:117] "RemoveContainer" containerID="22aa1191e56f80d24db5f1d591bcfec381396f4e4226b0a749a6f57b91eb041d"
Aug 01 00:24:23.500666 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.500177    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"
Aug 01 00:24:23.502084 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.502066    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"
Aug 01 00:24:23.502176 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.502096    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv" event={"ID":"69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff","Type":"ContainerDied","Data":"3af73639b90a4425d4eb2bee253525d773b345d2f593ade14f44fe2c2820fa60"}
Aug 01 00:24:23.511225 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.511210    2584 scope.go:117] "RemoveContainer" containerID="5b6ce5a160c7de77ddf8083ce28b518ccf921096ed1bdc9feda0d119bc3f5cbc"
Aug 01 00:24:23.518853 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.518838    2584 scope.go:117] "RemoveContainer" containerID="a1ac3610c4d776e255cde4b4babf6f6508d5193c350a38bc029be0c90c8be835"
Aug 01 00:24:23.526591 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.526576    2584 scope.go:117] "RemoveContainer" containerID="d5b37e224620d1e75fd56afabf6c8da34960a6630ab5d81e90505b39bf154565"
Aug 01 00:24:23.526829 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.526809    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"]
Aug 01 00:24:23.530899 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.530880    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-697f894f79-gmqgc"]
Aug 01 00:24:23.533867 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.533850    2584 scope.go:117] "RemoveContainer" containerID="72d72338b13db3e819a54561baa553b10b3c095d261672439c391c4421c5a71e"
Aug 01 00:24:23.540847 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.540832    2584 scope.go:117] "RemoveContainer" containerID="acde00a59bcbfcb6c48c26d23b431a63516e44a51570e84027c2c52ed2407ecc"
Aug 01 00:24:23.543630 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.543593    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"]
Aug 01 00:24:23.545368 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.545350    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-knppv"]
Aug 01 00:24:23.571711 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.571681    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" path="/var/lib/kubelet/pods/69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff/volumes"
Aug 01 00:24:23.572171 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:23.572158    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" path="/var/lib/kubelet/pods/f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa/volumes"
Aug 01 00:24:27.561308 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561272    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"]
Aug 01 00:24:27.561683 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561579    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kube-rbac-proxy"
Aug 01 00:24:27.561683 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561589    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kube-rbac-proxy"
Aug 01 00:24:27.561683 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561598    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="storage-initializer"
Aug 01 00:24:27.561683 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561605    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="storage-initializer"
Aug 01 00:24:27.561683 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561611    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kserve-container"
Aug 01 00:24:27.561683 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561617    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kserve-container"
Aug 01 00:24:27.561683 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561624    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kube-rbac-proxy"
Aug 01 00:24:27.561683 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561629    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kube-rbac-proxy"
Aug 01 00:24:27.561683 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561635    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="storage-initializer"
Aug 01 00:24:27.561683 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561641    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="storage-initializer"
Aug 01 00:24:27.561683 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561646    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kserve-container"
Aug 01 00:24:27.561683 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561651    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kserve-container"
Aug 01 00:24:27.562115 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561710    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kserve-container"
Aug 01 00:24:27.562115 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561719    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c5df2c-d17b-49cb-8ef8-e1fbff732bfa" containerName="kube-rbac-proxy"
Aug 01 00:24:27.562115 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561726    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kube-rbac-proxy"
Aug 01 00:24:27.562115 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.561731    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="69d8486c-0aa6-4f10-b5f0-e6efbe3a6eff" containerName="kserve-container"
Aug 01 00:24:27.566903 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.566877    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.569488 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.569455    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Aug 01 00:24:27.569488 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.569467    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:24:27.569674 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.569528    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Aug 01 00:24:27.569674 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.569556    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:24:27.570548 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.570522    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:24:27.570707 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.570690    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:24:27.570783 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.570726    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:24:27.575452 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.575431    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"]
Aug 01 00:24:27.628353 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.628320    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16161bf8-8365-4d93-944e-ce29721ba52e-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-bm4vg\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.628522 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.628363    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16161bf8-8365-4d93-944e-ce29721ba52e-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-bm4vg\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.628522 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.628420    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch52n\" (UniqueName: \"kubernetes.io/projected/16161bf8-8365-4d93-944e-ce29721ba52e-kube-api-access-ch52n\") pod \"isvc-lightgbm-predictor-54f85ddf88-bm4vg\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.628522 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.628455    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16161bf8-8365-4d93-944e-ce29721ba52e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-bm4vg\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.628522 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.628477    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16161bf8-8365-4d93-944e-ce29721ba52e-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-bm4vg\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.729096 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.729065    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16161bf8-8365-4d93-944e-ce29721ba52e-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-bm4vg\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.729278 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.729112    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16161bf8-8365-4d93-944e-ce29721ba52e-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-bm4vg\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.729278 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.729173    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch52n\" (UniqueName: \"kubernetes.io/projected/16161bf8-8365-4d93-944e-ce29721ba52e-kube-api-access-ch52n\") pod \"isvc-lightgbm-predictor-54f85ddf88-bm4vg\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.729278 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.729208    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16161bf8-8365-4d93-944e-ce29721ba52e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-bm4vg\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.729278 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.729236    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16161bf8-8365-4d93-944e-ce29721ba52e-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-bm4vg\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.729523 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.729501    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16161bf8-8365-4d93-944e-ce29721ba52e-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-bm4vg\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.729865 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.729844    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16161bf8-8365-4d93-944e-ce29721ba52e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-bm4vg\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.730022 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.730000    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16161bf8-8365-4d93-944e-ce29721ba52e-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-bm4vg\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.731648 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.731627    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16161bf8-8365-4d93-944e-ce29721ba52e-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-bm4vg\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.754293 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.754267    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch52n\" (UniqueName: \"kubernetes.io/projected/16161bf8-8365-4d93-944e-ce29721ba52e-kube-api-access-ch52n\") pod \"isvc-lightgbm-predictor-54f85ddf88-bm4vg\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:27.879455 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:27.879371    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:28.009787 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:28.009762    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"]
Aug 01 00:24:28.011649 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:24:28.011627    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16161bf8_8365_4d93_944e_ce29721ba52e.slice/crio-e00affcb623d3384de02f69cf2d7a5a2985fd873f6cf39d98a9a436234fc5067 WatchSource:0}: Error finding container e00affcb623d3384de02f69cf2d7a5a2985fd873f6cf39d98a9a436234fc5067: Status 404 returned error can't find the container with id e00affcb623d3384de02f69cf2d7a5a2985fd873f6cf39d98a9a436234fc5067
Aug 01 00:24:28.528872 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:28.528838    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" event={"ID":"16161bf8-8365-4d93-944e-ce29721ba52e","Type":"ContainerStarted","Data":"76bdb1557fea238dbd469a29148ad4165a4ecdddc012d97f36e2dac301b98187"}
Aug 01 00:24:28.529033 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:28.528895    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" event={"ID":"16161bf8-8365-4d93-944e-ce29721ba52e","Type":"ContainerStarted","Data":"e00affcb623d3384de02f69cf2d7a5a2985fd873f6cf39d98a9a436234fc5067"}
Aug 01 00:24:32.546501 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:32.546470    2584 generic.go:356] "Generic (PLEG): container finished" podID="16161bf8-8365-4d93-944e-ce29721ba52e" containerID="76bdb1557fea238dbd469a29148ad4165a4ecdddc012d97f36e2dac301b98187" exitCode=0
Aug 01 00:24:32.547004 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:32.546525    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" event={"ID":"16161bf8-8365-4d93-944e-ce29721ba52e","Type":"ContainerDied","Data":"76bdb1557fea238dbd469a29148ad4165a4ecdddc012d97f36e2dac301b98187"}
Aug 01 00:24:32.547677 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:32.547659    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:24:39.583534 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:39.583495    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" event={"ID":"16161bf8-8365-4d93-944e-ce29721ba52e","Type":"ContainerStarted","Data":"fe89ff282a50761685525e72b0a3547c325881e8abf0678470cde33b1d8247d1"}
Aug 01 00:24:39.584000 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:39.583558    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" event={"ID":"16161bf8-8365-4d93-944e-ce29721ba52e","Type":"ContainerStarted","Data":"fab7dc1bd14779184c992ec722ef7622a5bd76659c60e2754736ae4250466e10"}
Aug 01 00:24:39.584000 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:39.583721    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:39.603505 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:39.603449    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" podStartSLOduration=2.439147868 podStartE2EDuration="12.603433925s" totalImagesPullingTime="6.374013177s" totalInitContainerRuntime="3.79027288s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:24:39.602804971 +0000 UTC m=+2314.476448607" watchObservedRunningTime="2026-08-01 00:24:39.603433925 +0000 UTC m=+2314.477077535"
Aug 01 00:24:40.588826 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:40.588794    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:40.590011 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:40.589985    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:24:41.592298 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:41.592249    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:24:46.596472 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:46.596443    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:24:46.596955 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:46.596929    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:24:56.597367 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:24:56.597324    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:25:06.597890 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:25:06.597804    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:25:16.597338 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:25:16.597297    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:25:26.597500 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:25:26.597463    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:25:36.597246 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:25:36.597208    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:25:46.597368 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:25:46.597327    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:25:56.598126 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:25:56.598097    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:26:01.670653 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:01.670623    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"]
Aug 01 00:26:01.671058 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:01.670847    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kserve-container" containerID="cri-o://fab7dc1bd14779184c992ec722ef7622a5bd76659c60e2754736ae4250466e10" gracePeriod=30
Aug 01 00:26:01.671058 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:01.670882    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kube-rbac-proxy" containerID="cri-o://fe89ff282a50761685525e72b0a3547c325881e8abf0678470cde33b1d8247d1" gracePeriod=30
Aug 01 00:26:01.925425 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:01.925345    2584 generic.go:356] "Generic (PLEG): container finished" podID="16161bf8-8365-4d93-944e-ce29721ba52e" containerID="fe89ff282a50761685525e72b0a3547c325881e8abf0678470cde33b1d8247d1" exitCode=2
Aug 01 00:26:01.925425 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:01.925414    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" event={"ID":"16161bf8-8365-4d93-944e-ce29721ba52e","Type":"ContainerDied","Data":"fe89ff282a50761685525e72b0a3547c325881e8abf0678470cde33b1d8247d1"}
Aug 01 00:26:05.679907 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:05.679876    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:26:05.684666 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:05.684647    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:26:06.211560 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.211533    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:26:06.239754 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.239724    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch52n\" (UniqueName: \"kubernetes.io/projected/16161bf8-8365-4d93-944e-ce29721ba52e-kube-api-access-ch52n\") pod \"16161bf8-8365-4d93-944e-ce29721ba52e\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") "
Aug 01 00:26:06.239754 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.239760    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16161bf8-8365-4d93-944e-ce29721ba52e-proxy-tls\") pod \"16161bf8-8365-4d93-944e-ce29721ba52e\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") "
Aug 01 00:26:06.239985 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.239807    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16161bf8-8365-4d93-944e-ce29721ba52e-cabundle-cert\") pod \"16161bf8-8365-4d93-944e-ce29721ba52e\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") "
Aug 01 00:26:06.239985 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.239838    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16161bf8-8365-4d93-944e-ce29721ba52e-kube-rbac-proxy-sar-config\") pod \"16161bf8-8365-4d93-944e-ce29721ba52e\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") "
Aug 01 00:26:06.239985 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.239860    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16161bf8-8365-4d93-944e-ce29721ba52e-kserve-provision-location\") pod \"16161bf8-8365-4d93-944e-ce29721ba52e\" (UID: \"16161bf8-8365-4d93-944e-ce29721ba52e\") "
Aug 01 00:26:06.240244 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.240215    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16161bf8-8365-4d93-944e-ce29721ba52e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "16161bf8-8365-4d93-944e-ce29721ba52e" (UID: "16161bf8-8365-4d93-944e-ce29721ba52e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:26:06.240386 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.240239    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16161bf8-8365-4d93-944e-ce29721ba52e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "16161bf8-8365-4d93-944e-ce29721ba52e" (UID: "16161bf8-8365-4d93-944e-ce29721ba52e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:26:06.240386 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.240301    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16161bf8-8365-4d93-944e-ce29721ba52e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "16161bf8-8365-4d93-944e-ce29721ba52e" (UID: "16161bf8-8365-4d93-944e-ce29721ba52e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:26:06.241921 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.241897    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16161bf8-8365-4d93-944e-ce29721ba52e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "16161bf8-8365-4d93-944e-ce29721ba52e" (UID: "16161bf8-8365-4d93-944e-ce29721ba52e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:26:06.241921 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.241909    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16161bf8-8365-4d93-944e-ce29721ba52e-kube-api-access-ch52n" (OuterVolumeSpecName: "kube-api-access-ch52n") pod "16161bf8-8365-4d93-944e-ce29721ba52e" (UID: "16161bf8-8365-4d93-944e-ce29721ba52e"). InnerVolumeSpecName "kube-api-access-ch52n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:26:06.340550 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.340515    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch52n\" (UniqueName: \"kubernetes.io/projected/16161bf8-8365-4d93-944e-ce29721ba52e-kube-api-access-ch52n\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:26:06.340550 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.340544    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16161bf8-8365-4d93-944e-ce29721ba52e-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:26:06.340550 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.340554    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16161bf8-8365-4d93-944e-ce29721ba52e-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:26:06.340794 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.340563    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16161bf8-8365-4d93-944e-ce29721ba52e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:26:06.340794 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.340573    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16161bf8-8365-4d93-944e-ce29721ba52e-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:26:06.895319 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.895275    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"]
Aug 01 00:26:06.895719 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.895573    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kserve-container"
Aug 01 00:26:06.895719 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.895584    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kserve-container"
Aug 01 00:26:06.895719 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.895595    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kube-rbac-proxy"
Aug 01 00:26:06.895719 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.895600    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kube-rbac-proxy"
Aug 01 00:26:06.895719 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.895613    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="storage-initializer"
Aug 01 00:26:06.895719 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.895619    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="storage-initializer"
Aug 01 00:26:06.895719 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.895665    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kserve-container"
Aug 01 00:26:06.895719 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.895674    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" containerName="kube-rbac-proxy"
Aug 01 00:26:06.898854 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.898838    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:06.904475 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.904453    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Aug 01 00:26:06.905122 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.904913    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Aug 01 00:26:06.944502 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.944468    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkfr\" (UniqueName: \"kubernetes.io/projected/32186aef-ba87-4512-8101-2c8722433ce5-kube-api-access-trkfr\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:06.944668 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.944515    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32186aef-ba87-4512-8101-2c8722433ce5-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:06.944668 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.944542    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32186aef-ba87-4512-8101-2c8722433ce5-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:06.944668 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.944564    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32186aef-ba87-4512-8101-2c8722433ce5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:06.944668 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.944654    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32186aef-ba87-4512-8101-2c8722433ce5-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:06.947464 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.947432    2584 generic.go:356] "Generic (PLEG): container finished" podID="16161bf8-8365-4d93-944e-ce29721ba52e" containerID="fab7dc1bd14779184c992ec722ef7622a5bd76659c60e2754736ae4250466e10" exitCode=0
Aug 01 00:26:06.947615 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.947474    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" event={"ID":"16161bf8-8365-4d93-944e-ce29721ba52e","Type":"ContainerDied","Data":"fab7dc1bd14779184c992ec722ef7622a5bd76659c60e2754736ae4250466e10"}
Aug 01 00:26:06.947615 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.947505    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"
Aug 01 00:26:06.947615 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.947550    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg" event={"ID":"16161bf8-8365-4d93-944e-ce29721ba52e","Type":"ContainerDied","Data":"e00affcb623d3384de02f69cf2d7a5a2985fd873f6cf39d98a9a436234fc5067"}
Aug 01 00:26:06.947615 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.947591    2584 scope.go:117] "RemoveContainer" containerID="fe89ff282a50761685525e72b0a3547c325881e8abf0678470cde33b1d8247d1"
Aug 01 00:26:06.952277 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.952251    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"]
Aug 01 00:26:06.956927 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.956904    2584 scope.go:117] "RemoveContainer" containerID="fab7dc1bd14779184c992ec722ef7622a5bd76659c60e2754736ae4250466e10"
Aug 01 00:26:06.965766 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.965750    2584 scope.go:117] "RemoveContainer" containerID="76bdb1557fea238dbd469a29148ad4165a4ecdddc012d97f36e2dac301b98187"
Aug 01 00:26:06.972801 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.972785    2584 scope.go:117] "RemoveContainer" containerID="fe89ff282a50761685525e72b0a3547c325881e8abf0678470cde33b1d8247d1"
Aug 01 00:26:06.973065 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:26:06.973046    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe89ff282a50761685525e72b0a3547c325881e8abf0678470cde33b1d8247d1\": container with ID starting with fe89ff282a50761685525e72b0a3547c325881e8abf0678470cde33b1d8247d1 not found: ID does not exist" containerID="fe89ff282a50761685525e72b0a3547c325881e8abf0678470cde33b1d8247d1"
Aug 01 00:26:06.973119 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.973073    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe89ff282a50761685525e72b0a3547c325881e8abf0678470cde33b1d8247d1"} err="failed to get container status \"fe89ff282a50761685525e72b0a3547c325881e8abf0678470cde33b1d8247d1\": rpc error: code = NotFound desc = could not find container \"fe89ff282a50761685525e72b0a3547c325881e8abf0678470cde33b1d8247d1\": container with ID starting with fe89ff282a50761685525e72b0a3547c325881e8abf0678470cde33b1d8247d1 not found: ID does not exist"
Aug 01 00:26:06.973119 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.973092    2584 scope.go:117] "RemoveContainer" containerID="fab7dc1bd14779184c992ec722ef7622a5bd76659c60e2754736ae4250466e10"
Aug 01 00:26:06.973302 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:26:06.973287    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab7dc1bd14779184c992ec722ef7622a5bd76659c60e2754736ae4250466e10\": container with ID starting with fab7dc1bd14779184c992ec722ef7622a5bd76659c60e2754736ae4250466e10 not found: ID does not exist" containerID="fab7dc1bd14779184c992ec722ef7622a5bd76659c60e2754736ae4250466e10"
Aug 01 00:26:06.973348 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.973307    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab7dc1bd14779184c992ec722ef7622a5bd76659c60e2754736ae4250466e10"} err="failed to get container status \"fab7dc1bd14779184c992ec722ef7622a5bd76659c60e2754736ae4250466e10\": rpc error: code = NotFound desc = could not find container \"fab7dc1bd14779184c992ec722ef7622a5bd76659c60e2754736ae4250466e10\": container with ID starting with fab7dc1bd14779184c992ec722ef7622a5bd76659c60e2754736ae4250466e10 not found: ID does not exist"
Aug 01 00:26:06.973348 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.973319    2584 scope.go:117] "RemoveContainer" containerID="76bdb1557fea238dbd469a29148ad4165a4ecdddc012d97f36e2dac301b98187"
Aug 01 00:26:06.973496 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:26:06.973482    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76bdb1557fea238dbd469a29148ad4165a4ecdddc012d97f36e2dac301b98187\": container with ID starting with 76bdb1557fea238dbd469a29148ad4165a4ecdddc012d97f36e2dac301b98187 not found: ID does not exist" containerID="76bdb1557fea238dbd469a29148ad4165a4ecdddc012d97f36e2dac301b98187"
Aug 01 00:26:06.973539 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:06.973496    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76bdb1557fea238dbd469a29148ad4165a4ecdddc012d97f36e2dac301b98187"} err="failed to get container status \"76bdb1557fea238dbd469a29148ad4165a4ecdddc012d97f36e2dac301b98187\": rpc error: code = NotFound desc = could not find container \"76bdb1557fea238dbd469a29148ad4165a4ecdddc012d97f36e2dac301b98187\": container with ID starting with 76bdb1557fea238dbd469a29148ad4165a4ecdddc012d97f36e2dac301b98187 not found: ID does not exist"
Aug 01 00:26:07.036282 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.036246    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"]
Aug 01 00:26:07.046000 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.045951    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32186aef-ba87-4512-8101-2c8722433ce5-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:07.046141 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.046009    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32186aef-ba87-4512-8101-2c8722433ce5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:07.046141 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.046039    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32186aef-ba87-4512-8101-2c8722433ce5-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:07.046141 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.046090    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkfr\" (UniqueName: \"kubernetes.io/projected/32186aef-ba87-4512-8101-2c8722433ce5-kube-api-access-trkfr\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:07.046141 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.046117    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32186aef-ba87-4512-8101-2c8722433ce5-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:07.046364 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:26:07.046223    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Aug 01 00:26:07.046364 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:26:07.046278    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32186aef-ba87-4512-8101-2c8722433ce5-proxy-tls podName:32186aef-ba87-4512-8101-2c8722433ce5 nodeName:}" failed. No retries permitted until 2026-08-01 00:26:07.546259719 +0000 UTC m=+2402.419903310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/32186aef-ba87-4512-8101-2c8722433ce5-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" (UID: "32186aef-ba87-4512-8101-2c8722433ce5") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Aug 01 00:26:07.046525 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.046480    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32186aef-ba87-4512-8101-2c8722433ce5-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:07.046802 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.046784    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32186aef-ba87-4512-8101-2c8722433ce5-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:07.046841 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.046808    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32186aef-ba87-4512-8101-2c8722433ce5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:07.049553 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.049535    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-bm4vg"]
Aug 01 00:26:07.061437 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.061417    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkfr\" (UniqueName: \"kubernetes.io/projected/32186aef-ba87-4512-8101-2c8722433ce5-kube-api-access-trkfr\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:07.550653 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.550616    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32186aef-ba87-4512-8101-2c8722433ce5-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:07.553096 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.553069    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32186aef-ba87-4512-8101-2c8722433ce5-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:07.571351 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.571319    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16161bf8-8365-4d93-944e-ce29721ba52e" path="/var/lib/kubelet/pods/16161bf8-8365-4d93-944e-ce29721ba52e/volumes"
Aug 01 00:26:07.808340 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.808238    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:07.941194 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.941169    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"]
Aug 01 00:26:07.942741 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:26:07.942714    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32186aef_ba87_4512_8101_2c8722433ce5.slice/crio-f8b89d4f3d88d5c119a76101cede63b171b9a7eb6d1a6f291ea805f024f7e444 WatchSource:0}: Error finding container f8b89d4f3d88d5c119a76101cede63b171b9a7eb6d1a6f291ea805f024f7e444: Status 404 returned error can't find the container with id f8b89d4f3d88d5c119a76101cede63b171b9a7eb6d1a6f291ea805f024f7e444
Aug 01 00:26:07.953093 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:07.953067    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" event={"ID":"32186aef-ba87-4512-8101-2c8722433ce5","Type":"ContainerStarted","Data":"f8b89d4f3d88d5c119a76101cede63b171b9a7eb6d1a6f291ea805f024f7e444"}
Aug 01 00:26:08.959088 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:08.959054    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" event={"ID":"32186aef-ba87-4512-8101-2c8722433ce5","Type":"ContainerStarted","Data":"fe0e61e8238557299d7112c581195fe578cd9dc30e4078d56dfa4326dec88121"}
Aug 01 00:26:11.973214 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:11.973183    2584 generic.go:356] "Generic (PLEG): container finished" podID="32186aef-ba87-4512-8101-2c8722433ce5" containerID="fe0e61e8238557299d7112c581195fe578cd9dc30e4078d56dfa4326dec88121" exitCode=0
Aug 01 00:26:11.973591 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:11.973256    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" event={"ID":"32186aef-ba87-4512-8101-2c8722433ce5","Type":"ContainerDied","Data":"fe0e61e8238557299d7112c581195fe578cd9dc30e4078d56dfa4326dec88121"}
Aug 01 00:26:12.979901 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:12.979870    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" event={"ID":"32186aef-ba87-4512-8101-2c8722433ce5","Type":"ContainerStarted","Data":"cf1a74716918a9449903250a9700a01fce23e7a98188923595a47e24b9888201"}
Aug 01 00:26:12.980302 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:12.979930    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" event={"ID":"32186aef-ba87-4512-8101-2c8722433ce5","Type":"ContainerStarted","Data":"04f91717811d9571719f7f27201b2470b2f477e77a9876eb22677dd24bf2d673"}
Aug 01 00:26:12.980302 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:12.980141    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:12.998818 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:12.998773    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" podStartSLOduration=3.221598964 podStartE2EDuration="6.998757146s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.777158182s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:26:12.997634982 +0000 UTC m=+2407.871278594" watchObservedRunningTime="2026-08-01 00:26:12.998757146 +0000 UTC m=+2407.872400756"
Aug 01 00:26:13.984451 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:13.984418    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:13.985668 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:13.985638    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Aug 01 00:26:14.989213 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:14.989170    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Aug 01 00:26:19.993317 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:19.993288    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:26:19.993800 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:19.993755    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Aug 01 00:26:29.994104 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:29.994021    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Aug 01 00:26:39.994389 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:39.994344    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Aug 01 00:26:49.994252 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:49.994206    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Aug 01 00:26:59.994451 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:26:59.994414    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Aug 01 00:27:09.993957 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:09.993915    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Aug 01 00:27:19.993869 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:19.993823    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Aug 01 00:27:29.994854 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:29.994825    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:27:37.458401 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:37.458362    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"]
Aug 01 00:27:37.458921 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:37.458617    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kserve-container" containerID="cri-o://04f91717811d9571719f7f27201b2470b2f477e77a9876eb22677dd24bf2d673" gracePeriod=30
Aug 01 00:27:37.458921 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:37.458682    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kube-rbac-proxy" containerID="cri-o://cf1a74716918a9449903250a9700a01fce23e7a98188923595a47e24b9888201" gracePeriod=30
Aug 01 00:27:38.341355 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:38.341318    2584 generic.go:356] "Generic (PLEG): container finished" podID="32186aef-ba87-4512-8101-2c8722433ce5" containerID="cf1a74716918a9449903250a9700a01fce23e7a98188923595a47e24b9888201" exitCode=2
Aug 01 00:27:38.341522 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:38.341361    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" event={"ID":"32186aef-ba87-4512-8101-2c8722433ce5","Type":"ContainerDied","Data":"cf1a74716918a9449903250a9700a01fce23e7a98188923595a47e24b9888201"}
Aug 01 00:27:39.989980 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:39.989920    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Aug 01 00:27:39.994248 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:39.994221    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Aug 01 00:27:42.103592 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.103566    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:27:42.238611 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.238523    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32186aef-ba87-4512-8101-2c8722433ce5-kube-rbac-proxy-sar-config\") pod \"32186aef-ba87-4512-8101-2c8722433ce5\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") "
Aug 01 00:27:42.238611 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.238606    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32186aef-ba87-4512-8101-2c8722433ce5-proxy-tls\") pod \"32186aef-ba87-4512-8101-2c8722433ce5\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") "
Aug 01 00:27:42.238846 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.238634    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trkfr\" (UniqueName: \"kubernetes.io/projected/32186aef-ba87-4512-8101-2c8722433ce5-kube-api-access-trkfr\") pod \"32186aef-ba87-4512-8101-2c8722433ce5\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") "
Aug 01 00:27:42.238846 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.238658    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32186aef-ba87-4512-8101-2c8722433ce5-cabundle-cert\") pod \"32186aef-ba87-4512-8101-2c8722433ce5\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") "
Aug 01 00:27:42.238846 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.238691    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32186aef-ba87-4512-8101-2c8722433ce5-kserve-provision-location\") pod \"32186aef-ba87-4512-8101-2c8722433ce5\" (UID: \"32186aef-ba87-4512-8101-2c8722433ce5\") "
Aug 01 00:27:42.239062 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.238914    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32186aef-ba87-4512-8101-2c8722433ce5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32186aef-ba87-4512-8101-2c8722433ce5" (UID: "32186aef-ba87-4512-8101-2c8722433ce5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:27:42.239108 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.239055    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32186aef-ba87-4512-8101-2c8722433ce5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "32186aef-ba87-4512-8101-2c8722433ce5" (UID: "32186aef-ba87-4512-8101-2c8722433ce5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:27:42.239146 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.239099    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32186aef-ba87-4512-8101-2c8722433ce5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32186aef-ba87-4512-8101-2c8722433ce5" (UID: "32186aef-ba87-4512-8101-2c8722433ce5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:27:42.240759 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.240728    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32186aef-ba87-4512-8101-2c8722433ce5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32186aef-ba87-4512-8101-2c8722433ce5" (UID: "32186aef-ba87-4512-8101-2c8722433ce5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:27:42.240875 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.240779    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32186aef-ba87-4512-8101-2c8722433ce5-kube-api-access-trkfr" (OuterVolumeSpecName: "kube-api-access-trkfr") pod "32186aef-ba87-4512-8101-2c8722433ce5" (UID: "32186aef-ba87-4512-8101-2c8722433ce5"). InnerVolumeSpecName "kube-api-access-trkfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:27:42.339326 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.339290    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32186aef-ba87-4512-8101-2c8722433ce5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:27:42.339326 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.339319    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32186aef-ba87-4512-8101-2c8722433ce5-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:27:42.339326 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.339331    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trkfr\" (UniqueName: \"kubernetes.io/projected/32186aef-ba87-4512-8101-2c8722433ce5-kube-api-access-trkfr\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:27:42.339550 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.339344    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32186aef-ba87-4512-8101-2c8722433ce5-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:27:42.339550 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.339356    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32186aef-ba87-4512-8101-2c8722433ce5-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:27:42.361442 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.361407    2584 generic.go:356] "Generic (PLEG): container finished" podID="32186aef-ba87-4512-8101-2c8722433ce5" containerID="04f91717811d9571719f7f27201b2470b2f477e77a9876eb22677dd24bf2d673" exitCode=0
Aug 01 00:27:42.361573 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.361464    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"
Aug 01 00:27:42.361573 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.361488    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" event={"ID":"32186aef-ba87-4512-8101-2c8722433ce5","Type":"ContainerDied","Data":"04f91717811d9571719f7f27201b2470b2f477e77a9876eb22677dd24bf2d673"}
Aug 01 00:27:42.361573 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.361549    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc" event={"ID":"32186aef-ba87-4512-8101-2c8722433ce5","Type":"ContainerDied","Data":"f8b89d4f3d88d5c119a76101cede63b171b9a7eb6d1a6f291ea805f024f7e444"}
Aug 01 00:27:42.361682 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.361574    2584 scope.go:117] "RemoveContainer" containerID="cf1a74716918a9449903250a9700a01fce23e7a98188923595a47e24b9888201"
Aug 01 00:27:42.369790 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.369772    2584 scope.go:117] "RemoveContainer" containerID="04f91717811d9571719f7f27201b2470b2f477e77a9876eb22677dd24bf2d673"
Aug 01 00:27:42.376921 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.376906    2584 scope.go:117] "RemoveContainer" containerID="fe0e61e8238557299d7112c581195fe578cd9dc30e4078d56dfa4326dec88121"
Aug 01 00:27:42.384637 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.384597    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"]
Aug 01 00:27:42.386007 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.385989    2584 scope.go:117] "RemoveContainer" containerID="cf1a74716918a9449903250a9700a01fce23e7a98188923595a47e24b9888201"
Aug 01 00:27:42.386284 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:27:42.386262    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1a74716918a9449903250a9700a01fce23e7a98188923595a47e24b9888201\": container with ID starting with cf1a74716918a9449903250a9700a01fce23e7a98188923595a47e24b9888201 not found: ID does not exist" containerID="cf1a74716918a9449903250a9700a01fce23e7a98188923595a47e24b9888201"
Aug 01 00:27:42.386374 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.386290    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1a74716918a9449903250a9700a01fce23e7a98188923595a47e24b9888201"} err="failed to get container status \"cf1a74716918a9449903250a9700a01fce23e7a98188923595a47e24b9888201\": rpc error: code = NotFound desc = could not find container \"cf1a74716918a9449903250a9700a01fce23e7a98188923595a47e24b9888201\": container with ID starting with cf1a74716918a9449903250a9700a01fce23e7a98188923595a47e24b9888201 not found: ID does not exist"
Aug 01 00:27:42.386374 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.386309    2584 scope.go:117] "RemoveContainer" containerID="04f91717811d9571719f7f27201b2470b2f477e77a9876eb22677dd24bf2d673"
Aug 01 00:27:42.386565 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:27:42.386543    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f91717811d9571719f7f27201b2470b2f477e77a9876eb22677dd24bf2d673\": container with ID starting with 04f91717811d9571719f7f27201b2470b2f477e77a9876eb22677dd24bf2d673 not found: ID does not exist" containerID="04f91717811d9571719f7f27201b2470b2f477e77a9876eb22677dd24bf2d673"
Aug 01 00:27:42.386655 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.386569    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f91717811d9571719f7f27201b2470b2f477e77a9876eb22677dd24bf2d673"} err="failed to get container status \"04f91717811d9571719f7f27201b2470b2f477e77a9876eb22677dd24bf2d673\": rpc error: code = NotFound desc = could not find container \"04f91717811d9571719f7f27201b2470b2f477e77a9876eb22677dd24bf2d673\": container with ID starting with 04f91717811d9571719f7f27201b2470b2f477e77a9876eb22677dd24bf2d673 not found: ID does not exist"
Aug 01 00:27:42.386655 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.386584    2584 scope.go:117] "RemoveContainer" containerID="fe0e61e8238557299d7112c581195fe578cd9dc30e4078d56dfa4326dec88121"
Aug 01 00:27:42.386790 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.386772    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2mncc"]
Aug 01 00:27:42.386838 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:27:42.386784    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe0e61e8238557299d7112c581195fe578cd9dc30e4078d56dfa4326dec88121\": container with ID starting with fe0e61e8238557299d7112c581195fe578cd9dc30e4078d56dfa4326dec88121 not found: ID does not exist" containerID="fe0e61e8238557299d7112c581195fe578cd9dc30e4078d56dfa4326dec88121"
Aug 01 00:27:42.386838 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:42.386810    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0e61e8238557299d7112c581195fe578cd9dc30e4078d56dfa4326dec88121"} err="failed to get container status \"fe0e61e8238557299d7112c581195fe578cd9dc30e4078d56dfa4326dec88121\": rpc error: code = NotFound desc = could not find container \"fe0e61e8238557299d7112c581195fe578cd9dc30e4078d56dfa4326dec88121\": container with ID starting with fe0e61e8238557299d7112c581195fe578cd9dc30e4078d56dfa4326dec88121 not found: ID does not exist"
Aug 01 00:27:43.570804 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:27:43.570772    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32186aef-ba87-4512-8101-2c8722433ce5" path="/var/lib/kubelet/pods/32186aef-ba87-4512-8101-2c8722433ce5/volumes"
Aug 01 00:28:37.918205 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.918120    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"]
Aug 01 00:28:37.918781 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.918549    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kserve-container"
Aug 01 00:28:37.918781 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.918568    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kserve-container"
Aug 01 00:28:37.918781 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.918602    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kube-rbac-proxy"
Aug 01 00:28:37.918781 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.918612    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kube-rbac-proxy"
Aug 01 00:28:37.918781 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.918623    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="storage-initializer"
Aug 01 00:28:37.918781 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.918631    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="storage-initializer"
Aug 01 00:28:37.918781 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.918719    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kserve-container"
Aug 01 00:28:37.918781 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.918730    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="32186aef-ba87-4512-8101-2c8722433ce5" containerName="kube-rbac-proxy"
Aug 01 00:28:37.921914 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.921894    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:37.924470 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.924444    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:28:37.924633 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.924447    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:28:37.924633 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.924565    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Aug 01 00:28:37.924767 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.924632    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:28:37.924824 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.924766    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:28:37.925622 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.925605    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:28:37.925702 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.925666    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Aug 01 00:28:37.931251 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.931232    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"]
Aug 01 00:28:37.970458 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.970432    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbgll\" (UniqueName: \"kubernetes.io/projected/a6d0aec7-d9e9-42c8-9220-50964a23814f-kube-api-access-dbgll\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:37.970582 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.970460    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6d0aec7-d9e9-42c8-9220-50964a23814f-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:37.970582 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.970485    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6d0aec7-d9e9-42c8-9220-50964a23814f-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:37.970878 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.970577    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6d0aec7-d9e9-42c8-9220-50964a23814f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:37.970878 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:37.970618    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6d0aec7-d9e9-42c8-9220-50964a23814f-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:38.071817 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:38.071784    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbgll\" (UniqueName: \"kubernetes.io/projected/a6d0aec7-d9e9-42c8-9220-50964a23814f-kube-api-access-dbgll\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:38.071817 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:38.071817    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6d0aec7-d9e9-42c8-9220-50964a23814f-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:38.072042 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:38.071840    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6d0aec7-d9e9-42c8-9220-50964a23814f-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:38.072042 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:38.071875    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6d0aec7-d9e9-42c8-9220-50964a23814f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:38.072042 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:38.071907    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6d0aec7-d9e9-42c8-9220-50964a23814f-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:38.072162 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:28:38.072118    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-serving-cert: secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Aug 01 00:28:38.072203 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:28:38.072180    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6d0aec7-d9e9-42c8-9220-50964a23814f-proxy-tls podName:a6d0aec7-d9e9-42c8-9220-50964a23814f nodeName:}" failed. No retries permitted until 2026-08-01 00:28:38.57216365 +0000 UTC m=+2553.445807248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a6d0aec7-d9e9-42c8-9220-50964a23814f-proxy-tls") pod "isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf" (UID: "a6d0aec7-d9e9-42c8-9220-50964a23814f") : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Aug 01 00:28:38.072273 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:38.072249    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6d0aec7-d9e9-42c8-9220-50964a23814f-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:38.072670 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:38.072649    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6d0aec7-d9e9-42c8-9220-50964a23814f-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:38.072711 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:38.072698    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6d0aec7-d9e9-42c8-9220-50964a23814f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:38.080086 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:38.080055    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbgll\" (UniqueName: \"kubernetes.io/projected/a6d0aec7-d9e9-42c8-9220-50964a23814f-kube-api-access-dbgll\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:38.576014 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:38.575977    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6d0aec7-d9e9-42c8-9220-50964a23814f-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:38.578319 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:38.578301    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6d0aec7-d9e9-42c8-9220-50964a23814f-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:38.833138 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:38.833047    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:38.972714 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:38.972684    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"]
Aug 01 00:28:38.974565 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:28:38.974528    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d0aec7_d9e9_42c8_9220_50964a23814f.slice/crio-5f6f829232b9b2571969fd0e2e0728303a287b7043987a6f218af0d9f62dc3bb WatchSource:0}: Error finding container 5f6f829232b9b2571969fd0e2e0728303a287b7043987a6f218af0d9f62dc3bb: Status 404 returned error can't find the container with id 5f6f829232b9b2571969fd0e2e0728303a287b7043987a6f218af0d9f62dc3bb
Aug 01 00:28:39.606098 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:39.606063    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf" event={"ID":"a6d0aec7-d9e9-42c8-9220-50964a23814f","Type":"ContainerStarted","Data":"51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290"}
Aug 01 00:28:39.606298 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:39.606122    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf" event={"ID":"a6d0aec7-d9e9-42c8-9220-50964a23814f","Type":"ContainerStarted","Data":"5f6f829232b9b2571969fd0e2e0728303a287b7043987a6f218af0d9f62dc3bb"}
Aug 01 00:28:43.624653 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:43.624621    2584 generic.go:356] "Generic (PLEG): container finished" podID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerID="51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290" exitCode=0
Aug 01 00:28:43.625172 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:43.624700    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf" event={"ID":"a6d0aec7-d9e9-42c8-9220-50964a23814f","Type":"ContainerDied","Data":"51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290"}
Aug 01 00:28:44.631371 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:44.631338    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf" event={"ID":"a6d0aec7-d9e9-42c8-9220-50964a23814f","Type":"ContainerStarted","Data":"3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d"}
Aug 01 00:28:44.631814 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:44.631398    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf" event={"ID":"a6d0aec7-d9e9-42c8-9220-50964a23814f","Type":"ContainerStarted","Data":"630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b"}
Aug 01 00:28:44.631814 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:44.631715    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:44.631940 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:44.631866    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:44.633427 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:44.633399    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf" podUID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Aug 01 00:28:44.652429 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:44.652373    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf" podStartSLOduration=3.785203065 podStartE2EDuration="7.652358823s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.867155758s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:28:44.650655817 +0000 UTC m=+2559.524299427" watchObservedRunningTime="2026-08-01 00:28:44.652358823 +0000 UTC m=+2559.526002432"
Aug 01 00:28:45.635597 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:45.635546    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf" podUID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Aug 01 00:28:50.641986 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:50.641933    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:28:50.642509 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:28:50.642480    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf" podUID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Aug 01 00:29:00.643115 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:00.643078    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:29:07.980372 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:07.980336    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"]
Aug 01 00:29:07.980845 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:07.980619    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf" podUID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerName="kserve-container" containerID="cri-o://630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b" gracePeriod=30
Aug 01 00:29:07.980845 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:07.980687    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf" podUID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerName="kube-rbac-proxy" containerID="cri-o://3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d" gracePeriod=30
Aug 01 00:29:08.624744 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.624720    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:29:08.728554 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.728477    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6d0aec7-d9e9-42c8-9220-50964a23814f-kserve-provision-location\") pod \"a6d0aec7-d9e9-42c8-9220-50964a23814f\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") "
Aug 01 00:29:08.728712 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.728550    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6d0aec7-d9e9-42c8-9220-50964a23814f-proxy-tls\") pod \"a6d0aec7-d9e9-42c8-9220-50964a23814f\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") "
Aug 01 00:29:08.728712 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.728587    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6d0aec7-d9e9-42c8-9220-50964a23814f-cabundle-cert\") pod \"a6d0aec7-d9e9-42c8-9220-50964a23814f\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") "
Aug 01 00:29:08.728712 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.728628    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6d0aec7-d9e9-42c8-9220-50964a23814f-kube-rbac-proxy-sar-config\") pod \"a6d0aec7-d9e9-42c8-9220-50964a23814f\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") "
Aug 01 00:29:08.728712 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.728658    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbgll\" (UniqueName: \"kubernetes.io/projected/a6d0aec7-d9e9-42c8-9220-50964a23814f-kube-api-access-dbgll\") pod \"a6d0aec7-d9e9-42c8-9220-50964a23814f\" (UID: \"a6d0aec7-d9e9-42c8-9220-50964a23814f\") "
Aug 01 00:29:08.728932 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.728898    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d0aec7-d9e9-42c8-9220-50964a23814f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a6d0aec7-d9e9-42c8-9220-50964a23814f" (UID: "a6d0aec7-d9e9-42c8-9220-50964a23814f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:29:08.729079 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.728976    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d0aec7-d9e9-42c8-9220-50964a23814f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a6d0aec7-d9e9-42c8-9220-50964a23814f" (UID: "a6d0aec7-d9e9-42c8-9220-50964a23814f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:29:08.729079 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.729008    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d0aec7-d9e9-42c8-9220-50964a23814f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a6d0aec7-d9e9-42c8-9220-50964a23814f" (UID: "a6d0aec7-d9e9-42c8-9220-50964a23814f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:29:08.730707 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.730682    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d0aec7-d9e9-42c8-9220-50964a23814f-kube-api-access-dbgll" (OuterVolumeSpecName: "kube-api-access-dbgll") pod "a6d0aec7-d9e9-42c8-9220-50964a23814f" (UID: "a6d0aec7-d9e9-42c8-9220-50964a23814f"). InnerVolumeSpecName "kube-api-access-dbgll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:29:08.730817 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.730714    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d0aec7-d9e9-42c8-9220-50964a23814f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a6d0aec7-d9e9-42c8-9220-50964a23814f" (UID: "a6d0aec7-d9e9-42c8-9220-50964a23814f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:29:08.738188 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.738164    2584 generic.go:356] "Generic (PLEG): container finished" podID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerID="3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d" exitCode=2
Aug 01 00:29:08.738188 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.738187    2584 generic.go:356] "Generic (PLEG): container finished" podID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerID="630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b" exitCode=0
Aug 01 00:29:08.738333 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.738232    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf" event={"ID":"a6d0aec7-d9e9-42c8-9220-50964a23814f","Type":"ContainerDied","Data":"3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d"}
Aug 01 00:29:08.738333 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.738288    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf" event={"ID":"a6d0aec7-d9e9-42c8-9220-50964a23814f","Type":"ContainerDied","Data":"630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b"}
Aug 01 00:29:08.738333 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.738307    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"
Aug 01 00:29:08.738333 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.738322    2584 scope.go:117] "RemoveContainer" containerID="3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d"
Aug 01 00:29:08.738475 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.738313    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf" event={"ID":"a6d0aec7-d9e9-42c8-9220-50964a23814f","Type":"ContainerDied","Data":"5f6f829232b9b2571969fd0e2e0728303a287b7043987a6f218af0d9f62dc3bb"}
Aug 01 00:29:08.745897 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.745794    2584 scope.go:117] "RemoveContainer" containerID="630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b"
Aug 01 00:29:08.753072 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.753055    2584 scope.go:117] "RemoveContainer" containerID="51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290"
Aug 01 00:29:08.759858 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.759836    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"]
Aug 01 00:29:08.760233 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.760212    2584 scope.go:117] "RemoveContainer" containerID="3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d"
Aug 01 00:29:08.760507 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:29:08.760490    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d\": container with ID starting with 3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d not found: ID does not exist" containerID="3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d"
Aug 01 00:29:08.760562 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.760516    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d"} err="failed to get container status \"3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d\": rpc error: code = NotFound desc = could not find container \"3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d\": container with ID starting with 3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d not found: ID does not exist"
Aug 01 00:29:08.760562 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.760532    2584 scope.go:117] "RemoveContainer" containerID="630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b"
Aug 01 00:29:08.760739 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:29:08.760722    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b\": container with ID starting with 630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b not found: ID does not exist" containerID="630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b"
Aug 01 00:29:08.760803 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.760749    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b"} err="failed to get container status \"630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b\": rpc error: code = NotFound desc = could not find container \"630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b\": container with ID starting with 630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b not found: ID does not exist"
Aug 01 00:29:08.760803 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.760770    2584 scope.go:117] "RemoveContainer" containerID="51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290"
Aug 01 00:29:08.761027 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:29:08.761012    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290\": container with ID starting with 51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290 not found: ID does not exist" containerID="51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290"
Aug 01 00:29:08.761082 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.761031    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290"} err="failed to get container status \"51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290\": rpc error: code = NotFound desc = could not find container \"51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290\": container with ID starting with 51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290 not found: ID does not exist"
Aug 01 00:29:08.761082 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.761044    2584 scope.go:117] "RemoveContainer" containerID="3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d"
Aug 01 00:29:08.761241 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.761221    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d"} err="failed to get container status \"3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d\": rpc error: code = NotFound desc = could not find container \"3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d\": container with ID starting with 3959c543fc01daa575c97f57a9dc7e5a0c03859e5842aeab95459769ccbbb53d not found: ID does not exist"
Aug 01 00:29:08.761313 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.761242    2584 scope.go:117] "RemoveContainer" containerID="630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b"
Aug 01 00:29:08.761482 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.761464    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b"} err="failed to get container status \"630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b\": rpc error: code = NotFound desc = could not find container \"630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b\": container with ID starting with 630636a0dbc5712fbd5ce48d2f78c11944ddce9b0ee4518f1f2026e53bcf688b not found: ID does not exist"
Aug 01 00:29:08.761540 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.761484    2584 scope.go:117] "RemoveContainer" containerID="51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290"
Aug 01 00:29:08.761685 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.761668    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290"} err="failed to get container status \"51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290\": rpc error: code = NotFound desc = could not find container \"51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290\": container with ID starting with 51cebc44af4defeacf441a64330bee83ee2fbc49443c69e479945fefe87ec290 not found: ID does not exist"
Aug 01 00:29:08.764188 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.764168    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wnkcf"]
Aug 01 00:29:08.829866 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.829838    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6d0aec7-d9e9-42c8-9220-50964a23814f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:29:08.829866 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.829864    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbgll\" (UniqueName: \"kubernetes.io/projected/a6d0aec7-d9e9-42c8-9220-50964a23814f-kube-api-access-dbgll\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:29:08.830078 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.829874    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6d0aec7-d9e9-42c8-9220-50964a23814f-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:29:08.830078 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.829883    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6d0aec7-d9e9-42c8-9220-50964a23814f-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:29:08.830078 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:08.829914    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6d0aec7-d9e9-42c8-9220-50964a23814f-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:29:09.570417 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:29:09.570383    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6d0aec7-d9e9-42c8-9220-50964a23814f" path="/var/lib/kubelet/pods/a6d0aec7-d9e9-42c8-9220-50964a23814f/volumes"
Aug 01 00:30:08.451510 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.451473    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"]
Aug 01 00:30:08.451928 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.451760    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerName="kube-rbac-proxy"
Aug 01 00:30:08.451928 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.451772    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerName="kube-rbac-proxy"
Aug 01 00:30:08.451928 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.451784    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerName="kserve-container"
Aug 01 00:30:08.451928 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.451790    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerName="kserve-container"
Aug 01 00:30:08.451928 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.451801    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerName="storage-initializer"
Aug 01 00:30:08.451928 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.451807    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerName="storage-initializer"
Aug 01 00:30:08.451928 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.451859    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerName="kube-rbac-proxy"
Aug 01 00:30:08.451928 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.451866    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6d0aec7-d9e9-42c8-9220-50964a23814f" containerName="kserve-container"
Aug 01 00:30:08.454666 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.454647    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:08.457075 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.457050    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Aug 01 00:30:08.457563 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.457531    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:30:08.457563 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.457558    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:30:08.457730 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.457603    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:30:08.457730 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.457604    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:30:08.458167 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.458151    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:30:08.458214 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.458155    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Aug 01 00:30:08.465139 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.465117    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"]
Aug 01 00:30:08.509780 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.509744    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztx8t\" (UniqueName: \"kubernetes.io/projected/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kube-api-access-ztx8t\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:08.509780 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.509783    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e56f66de-e025-4eeb-9f7f-fac9c1152c50-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:08.510023 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.509813    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e56f66de-e025-4eeb-9f7f-fac9c1152c50-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:08.510023 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.509834    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:08.510023 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.509853    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:08.610861 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.610810    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztx8t\" (UniqueName: \"kubernetes.io/projected/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kube-api-access-ztx8t\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:08.611088 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.610898    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e56f66de-e025-4eeb-9f7f-fac9c1152c50-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:08.611088 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.610949    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e56f66de-e025-4eeb-9f7f-fac9c1152c50-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:08.611225 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:30:08.611099    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Aug 01 00:30:08.611225 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:30:08.611176    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e56f66de-e025-4eeb-9f7f-fac9c1152c50-proxy-tls podName:e56f66de-e025-4eeb-9f7f-fac9c1152c50 nodeName:}" failed. No retries permitted until 2026-08-01 00:30:09.11115424 +0000 UTC m=+2643.984797849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e56f66de-e025-4eeb-9f7f-fac9c1152c50-proxy-tls") pod "isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" (UID: "e56f66de-e025-4eeb-9f7f-fac9c1152c50") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Aug 01 00:30:08.611225 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.611200    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:08.611391 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.611234    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:08.611554 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.611533    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:08.611768 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.611748    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:08.611825 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.611751    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e56f66de-e025-4eeb-9f7f-fac9c1152c50-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:08.619163 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:08.619140    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztx8t\" (UniqueName: \"kubernetes.io/projected/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kube-api-access-ztx8t\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:09.114317 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:09.114259    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e56f66de-e025-4eeb-9f7f-fac9c1152c50-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:09.116747 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:09.116717    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e56f66de-e025-4eeb-9f7f-fac9c1152c50-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:09.366320 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:09.366223    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:09.498342 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:09.498313    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"]
Aug 01 00:30:09.500047 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:30:09.500020    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56f66de_e025_4eeb_9f7f_fac9c1152c50.slice/crio-9475eccfe9080338c6e850929d4bcbf51de54005d0f0e296fe11a513af2d76b1 WatchSource:0}: Error finding container 9475eccfe9080338c6e850929d4bcbf51de54005d0f0e296fe11a513af2d76b1: Status 404 returned error can't find the container with id 9475eccfe9080338c6e850929d4bcbf51de54005d0f0e296fe11a513af2d76b1
Aug 01 00:30:09.983895 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:09.983850    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" event={"ID":"e56f66de-e025-4eeb-9f7f-fac9c1152c50","Type":"ContainerStarted","Data":"7f90624f6e9b70000327d87316a336ec503505c2d6b9bc8bb578a32a4b0a68d7"}
Aug 01 00:30:09.984139 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:09.983926    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" event={"ID":"e56f66de-e025-4eeb-9f7f-fac9c1152c50","Type":"ContainerStarted","Data":"9475eccfe9080338c6e850929d4bcbf51de54005d0f0e296fe11a513af2d76b1"}
Aug 01 00:30:14.002502 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:14.002461    2584 generic.go:356] "Generic (PLEG): container finished" podID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerID="7f90624f6e9b70000327d87316a336ec503505c2d6b9bc8bb578a32a4b0a68d7" exitCode=0
Aug 01 00:30:14.002900 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:14.002536    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" event={"ID":"e56f66de-e025-4eeb-9f7f-fac9c1152c50","Type":"ContainerDied","Data":"7f90624f6e9b70000327d87316a336ec503505c2d6b9bc8bb578a32a4b0a68d7"}
Aug 01 00:30:14.061949 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:14.061930    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:30:15.011883 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:15.011747    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" event={"ID":"e56f66de-e025-4eeb-9f7f-fac9c1152c50","Type":"ContainerStarted","Data":"e5463f13fd8378962a06ed781ce511e43f213588998d2ef8dd8ca0e4921c7f4d"}
Aug 01 00:30:17.026039 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:17.026003    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" event={"ID":"e56f66de-e025-4eeb-9f7f-fac9c1152c50","Type":"ContainerStarted","Data":"2677c51e146371b6ddd805a3d5ae2dfadbbab0fbfbd416614cd401b23e0ffc15"}
Aug 01 00:30:17.026612 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:17.026583    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:17.026753 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:17.026643    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:17.026753 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:17.026669    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" event={"ID":"e56f66de-e025-4eeb-9f7f-fac9c1152c50","Type":"ContainerStarted","Data":"addc1cb449f57723390af84bd6adbc5095b39bf62bab4cc86c148363d9dd4452"}
Aug 01 00:30:17.052008 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:17.051946    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" podStartSLOduration=2.5797560859999997 podStartE2EDuration="9.051931203s" totalImagesPullingTime="2.811518962s" totalInitContainerRuntime="3.660656155s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:17.05081441 +0000 UTC m=+2651.924458041" watchObservedRunningTime="2026-08-01 00:30:17.051931203 +0000 UTC m=+2651.925574812"
Aug 01 00:30:18.031607 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:18.031582    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:24.043302 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:24.043275    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:30:44.044949 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:44.044906    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.46:8080: connect: connection refused"
Aug 01 00:30:49.367246 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:49.367202    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.46:8080: connect: connection refused"
Aug 01 00:30:54.047287 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:30:54.047254    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:31:05.705313 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:05.705231    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:31:05.710375 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:05.710352    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:31:24.048302 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:24.048270    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:31:28.494853 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:28.494808    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"]
Aug 01 00:31:28.495339 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:28.495076    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kserve-container" containerID="cri-o://e5463f13fd8378962a06ed781ce511e43f213588998d2ef8dd8ca0e4921c7f4d" gracePeriod=30
Aug 01 00:31:28.495339 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:28.495111    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kube-rbac-proxy" containerID="cri-o://2677c51e146371b6ddd805a3d5ae2dfadbbab0fbfbd416614cd401b23e0ffc15" gracePeriod=30
Aug 01 00:31:28.495339 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:28.495114    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kserve-agent" containerID="cri-o://addc1cb449f57723390af84bd6adbc5095b39bf62bab4cc86c148363d9dd4452" gracePeriod=30
Aug 01 00:31:29.037088 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:29.037039    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Aug 01 00:31:29.343624 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:29.343593    2584 generic.go:356] "Generic (PLEG): container finished" podID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerID="2677c51e146371b6ddd805a3d5ae2dfadbbab0fbfbd416614cd401b23e0ffc15" exitCode=2
Aug 01 00:31:29.343624 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:29.343630    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" event={"ID":"e56f66de-e025-4eeb-9f7f-fac9c1152c50","Type":"ContainerDied","Data":"2677c51e146371b6ddd805a3d5ae2dfadbbab0fbfbd416614cd401b23e0ffc15"}
Aug 01 00:31:31.354532 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:31.354501    2584 generic.go:356] "Generic (PLEG): container finished" podID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerID="e5463f13fd8378962a06ed781ce511e43f213588998d2ef8dd8ca0e4921c7f4d" exitCode=0
Aug 01 00:31:31.354917 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:31.354587    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" event={"ID":"e56f66de-e025-4eeb-9f7f-fac9c1152c50","Type":"ContainerDied","Data":"e5463f13fd8378962a06ed781ce511e43f213588998d2ef8dd8ca0e4921c7f4d"}
Aug 01 00:31:34.037860 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:34.037821    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Aug 01 00:31:34.044335 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:34.044310    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.46:8080: connect: connection refused"
Aug 01 00:31:39.037372 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:39.037252    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Aug 01 00:31:39.038613 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:39.037392    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:31:44.037606 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:44.037556    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Aug 01 00:31:44.044539 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:44.044495    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.46:8080: connect: connection refused"
Aug 01 00:31:49.037957 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:49.037916    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Aug 01 00:31:54.037155 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:54.037103    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Aug 01 00:31:54.044007 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:54.043945    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.46:8080: connect: connection refused"
Aug 01 00:31:54.044153 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:54.044090    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:31:58.696070 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.696043    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:31:58.728376 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.728349    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kube-rbac-proxy-sar-config\") pod \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") "
Aug 01 00:31:58.728542 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.728396    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztx8t\" (UniqueName: \"kubernetes.io/projected/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kube-api-access-ztx8t\") pod \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") "
Aug 01 00:31:58.728542 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.728419    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kserve-provision-location\") pod \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") "
Aug 01 00:31:58.728542 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.728449    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e56f66de-e025-4eeb-9f7f-fac9c1152c50-proxy-tls\") pod \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") "
Aug 01 00:31:58.728542 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.728483    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e56f66de-e025-4eeb-9f7f-fac9c1152c50-cabundle-cert\") pod \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\" (UID: \"e56f66de-e025-4eeb-9f7f-fac9c1152c50\") "
Aug 01 00:31:58.728768 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.728702    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e56f66de-e025-4eeb-9f7f-fac9c1152c50" (UID: "e56f66de-e025-4eeb-9f7f-fac9c1152c50"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:31:58.728921 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.728895    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56f66de-e025-4eeb-9f7f-fac9c1152c50-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e56f66de-e025-4eeb-9f7f-fac9c1152c50" (UID: "e56f66de-e025-4eeb-9f7f-fac9c1152c50"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:31:58.729126 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.729100    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e56f66de-e025-4eeb-9f7f-fac9c1152c50" (UID: "e56f66de-e025-4eeb-9f7f-fac9c1152c50"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:31:58.730576 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.730551    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kube-api-access-ztx8t" (OuterVolumeSpecName: "kube-api-access-ztx8t") pod "e56f66de-e025-4eeb-9f7f-fac9c1152c50" (UID: "e56f66de-e025-4eeb-9f7f-fac9c1152c50"). InnerVolumeSpecName "kube-api-access-ztx8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:31:58.730576 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.730583    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56f66de-e025-4eeb-9f7f-fac9c1152c50-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e56f66de-e025-4eeb-9f7f-fac9c1152c50" (UID: "e56f66de-e025-4eeb-9f7f-fac9c1152c50"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:31:58.830038 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.829987    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztx8t\" (UniqueName: \"kubernetes.io/projected/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kube-api-access-ztx8t\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:31:58.830038 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.830034    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:31:58.830038 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.830047    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e56f66de-e025-4eeb-9f7f-fac9c1152c50-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:31:58.830297 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.830060    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e56f66de-e025-4eeb-9f7f-fac9c1152c50-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:31:58.830297 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:58.830072    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e56f66de-e025-4eeb-9f7f-fac9c1152c50-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:31:59.481460 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.481412    2584 generic.go:356] "Generic (PLEG): container finished" podID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerID="addc1cb449f57723390af84bd6adbc5095b39bf62bab4cc86c148363d9dd4452" exitCode=137
Aug 01 00:31:59.481646 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.481471    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"
Aug 01 00:31:59.481646 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.481496    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" event={"ID":"e56f66de-e025-4eeb-9f7f-fac9c1152c50","Type":"ContainerDied","Data":"addc1cb449f57723390af84bd6adbc5095b39bf62bab4cc86c148363d9dd4452"}
Aug 01 00:31:59.481646 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.481565    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6" event={"ID":"e56f66de-e025-4eeb-9f7f-fac9c1152c50","Type":"ContainerDied","Data":"9475eccfe9080338c6e850929d4bcbf51de54005d0f0e296fe11a513af2d76b1"}
Aug 01 00:31:59.481646 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.481598    2584 scope.go:117] "RemoveContainer" containerID="2677c51e146371b6ddd805a3d5ae2dfadbbab0fbfbd416614cd401b23e0ffc15"
Aug 01 00:31:59.490484 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.490463    2584 scope.go:117] "RemoveContainer" containerID="addc1cb449f57723390af84bd6adbc5095b39bf62bab4cc86c148363d9dd4452"
Aug 01 00:31:59.497777 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.497757    2584 scope.go:117] "RemoveContainer" containerID="e5463f13fd8378962a06ed781ce511e43f213588998d2ef8dd8ca0e4921c7f4d"
Aug 01 00:31:59.503793 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.503772    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"]
Aug 01 00:31:59.506168 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.506015    2584 scope.go:117] "RemoveContainer" containerID="7f90624f6e9b70000327d87316a336ec503505c2d6b9bc8bb578a32a4b0a68d7"
Aug 01 00:31:59.506706 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.506682    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7cf87fd674-q6fk6"]
Aug 01 00:31:59.513468 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.513451    2584 scope.go:117] "RemoveContainer" containerID="2677c51e146371b6ddd805a3d5ae2dfadbbab0fbfbd416614cd401b23e0ffc15"
Aug 01 00:31:59.513735 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:31:59.513715    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2677c51e146371b6ddd805a3d5ae2dfadbbab0fbfbd416614cd401b23e0ffc15\": container with ID starting with 2677c51e146371b6ddd805a3d5ae2dfadbbab0fbfbd416614cd401b23e0ffc15 not found: ID does not exist" containerID="2677c51e146371b6ddd805a3d5ae2dfadbbab0fbfbd416614cd401b23e0ffc15"
Aug 01 00:31:59.513810 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.513757    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2677c51e146371b6ddd805a3d5ae2dfadbbab0fbfbd416614cd401b23e0ffc15"} err="failed to get container status \"2677c51e146371b6ddd805a3d5ae2dfadbbab0fbfbd416614cd401b23e0ffc15\": rpc error: code = NotFound desc = could not find container \"2677c51e146371b6ddd805a3d5ae2dfadbbab0fbfbd416614cd401b23e0ffc15\": container with ID starting with 2677c51e146371b6ddd805a3d5ae2dfadbbab0fbfbd416614cd401b23e0ffc15 not found: ID does not exist"
Aug 01 00:31:59.513810 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.513786    2584 scope.go:117] "RemoveContainer" containerID="addc1cb449f57723390af84bd6adbc5095b39bf62bab4cc86c148363d9dd4452"
Aug 01 00:31:59.514073 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:31:59.514056    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addc1cb449f57723390af84bd6adbc5095b39bf62bab4cc86c148363d9dd4452\": container with ID starting with addc1cb449f57723390af84bd6adbc5095b39bf62bab4cc86c148363d9dd4452 not found: ID does not exist" containerID="addc1cb449f57723390af84bd6adbc5095b39bf62bab4cc86c148363d9dd4452"
Aug 01 00:31:59.514137 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.514079    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addc1cb449f57723390af84bd6adbc5095b39bf62bab4cc86c148363d9dd4452"} err="failed to get container status \"addc1cb449f57723390af84bd6adbc5095b39bf62bab4cc86c148363d9dd4452\": rpc error: code = NotFound desc = could not find container \"addc1cb449f57723390af84bd6adbc5095b39bf62bab4cc86c148363d9dd4452\": container with ID starting with addc1cb449f57723390af84bd6adbc5095b39bf62bab4cc86c148363d9dd4452 not found: ID does not exist"
Aug 01 00:31:59.514137 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.514096    2584 scope.go:117] "RemoveContainer" containerID="e5463f13fd8378962a06ed781ce511e43f213588998d2ef8dd8ca0e4921c7f4d"
Aug 01 00:31:59.514314 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:31:59.514297    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5463f13fd8378962a06ed781ce511e43f213588998d2ef8dd8ca0e4921c7f4d\": container with ID starting with e5463f13fd8378962a06ed781ce511e43f213588998d2ef8dd8ca0e4921c7f4d not found: ID does not exist" containerID="e5463f13fd8378962a06ed781ce511e43f213588998d2ef8dd8ca0e4921c7f4d"
Aug 01 00:31:59.514358 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.514317    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5463f13fd8378962a06ed781ce511e43f213588998d2ef8dd8ca0e4921c7f4d"} err="failed to get container status \"e5463f13fd8378962a06ed781ce511e43f213588998d2ef8dd8ca0e4921c7f4d\": rpc error: code = NotFound desc = could not find container \"e5463f13fd8378962a06ed781ce511e43f213588998d2ef8dd8ca0e4921c7f4d\": container with ID starting with e5463f13fd8378962a06ed781ce511e43f213588998d2ef8dd8ca0e4921c7f4d not found: ID does not exist"
Aug 01 00:31:59.514358 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.514330    2584 scope.go:117] "RemoveContainer" containerID="7f90624f6e9b70000327d87316a336ec503505c2d6b9bc8bb578a32a4b0a68d7"
Aug 01 00:31:59.514515 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:31:59.514500    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f90624f6e9b70000327d87316a336ec503505c2d6b9bc8bb578a32a4b0a68d7\": container with ID starting with 7f90624f6e9b70000327d87316a336ec503505c2d6b9bc8bb578a32a4b0a68d7 not found: ID does not exist" containerID="7f90624f6e9b70000327d87316a336ec503505c2d6b9bc8bb578a32a4b0a68d7"
Aug 01 00:31:59.514555 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.514519    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f90624f6e9b70000327d87316a336ec503505c2d6b9bc8bb578a32a4b0a68d7"} err="failed to get container status \"7f90624f6e9b70000327d87316a336ec503505c2d6b9bc8bb578a32a4b0a68d7\": rpc error: code = NotFound desc = could not find container \"7f90624f6e9b70000327d87316a336ec503505c2d6b9bc8bb578a32a4b0a68d7\": container with ID starting with 7f90624f6e9b70000327d87316a336ec503505c2d6b9bc8bb578a32a4b0a68d7 not found: ID does not exist"
Aug 01 00:31:59.570285 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:31:59.570254    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" path="/var/lib/kubelet/pods/e56f66de-e025-4eeb-9f7f-fac9c1152c50/volumes"
Aug 01 00:32:50.533936 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.533897    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"]
Aug 01 00:32:50.534463 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.534249    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kserve-agent"
Aug 01 00:32:50.534463 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.534263    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kserve-agent"
Aug 01 00:32:50.534463 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.534274    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kube-rbac-proxy"
Aug 01 00:32:50.534463 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.534280    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kube-rbac-proxy"
Aug 01 00:32:50.534463 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.534287    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="storage-initializer"
Aug 01 00:32:50.534463 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.534294    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="storage-initializer"
Aug 01 00:32:50.534463 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.534302    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kserve-container"
Aug 01 00:32:50.534463 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.534310    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kserve-container"
Aug 01 00:32:50.534463 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.534402    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kube-rbac-proxy"
Aug 01 00:32:50.534463 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.534416    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kserve-agent"
Aug 01 00:32:50.534463 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.534423    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e56f66de-e025-4eeb-9f7f-fac9c1152c50" containerName="kserve-container"
Aug 01 00:32:50.537539 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.537521    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.540115 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.540088    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:32:50.540115 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.540105    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:32:50.540297 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.540109    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:32:50.540297 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.540185    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:32:50.541221 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.541190    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Aug 01 00:32:50.541327 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.541231    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Aug 01 00:32:50.541327 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.541241    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:32:50.546891 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.546871    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"]
Aug 01 00:32:50.653306 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.653276    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ca9cc3-b80d-4f0b-b941-a93c18813f86-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p4g59\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.653489 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.653327    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p4g59\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.653489 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.653346    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41ca9cc3-b80d-4f0b-b941-a93c18813f86-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p4g59\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.653489 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.653436    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b8n5\" (UniqueName: \"kubernetes.io/projected/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kube-api-access-2b8n5\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p4g59\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.653489 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.653467    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p4g59\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.753917 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.753877    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b8n5\" (UniqueName: \"kubernetes.io/projected/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kube-api-access-2b8n5\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p4g59\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.753917 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.753916    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p4g59\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.754157 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.754000    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ca9cc3-b80d-4f0b-b941-a93c18813f86-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p4g59\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.754157 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.754052    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p4g59\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.754259 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.754173    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41ca9cc3-b80d-4f0b-b941-a93c18813f86-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p4g59\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.754469 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.754446    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p4g59\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.754727 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.754706    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p4g59\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.754837 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.754816    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41ca9cc3-b80d-4f0b-b941-a93c18813f86-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p4g59\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.756610 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.756591    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ca9cc3-b80d-4f0b-b941-a93c18813f86-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p4g59\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.762318 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.762295    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b8n5\" (UniqueName: \"kubernetes.io/projected/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kube-api-access-2b8n5\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p4g59\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.849279 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.849247    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:32:50.976220 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:50.976174    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"]
Aug 01 00:32:50.977597 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:32:50.977557    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ca9cc3_b80d_4f0b_b941_a93c18813f86.slice/crio-2782029c3b793af55a2c29e6fad52c2babf9ddf6db19ac682bbedacaba8404ff WatchSource:0}: Error finding container 2782029c3b793af55a2c29e6fad52c2babf9ddf6db19ac682bbedacaba8404ff: Status 404 returned error can't find the container with id 2782029c3b793af55a2c29e6fad52c2babf9ddf6db19ac682bbedacaba8404ff
Aug 01 00:32:51.700875 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:51.700840    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" event={"ID":"41ca9cc3-b80d-4f0b-b941-a93c18813f86","Type":"ContainerStarted","Data":"dc9beb25f9cd838dd7021fca434af04377a2cbf99e96abc42948bf738c06e756"}
Aug 01 00:32:51.701386 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:51.700905    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" event={"ID":"41ca9cc3-b80d-4f0b-b941-a93c18813f86","Type":"ContainerStarted","Data":"2782029c3b793af55a2c29e6fad52c2babf9ddf6db19ac682bbedacaba8404ff"}
Aug 01 00:32:56.722317 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:56.722281    2584 generic.go:356] "Generic (PLEG): container finished" podID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerID="dc9beb25f9cd838dd7021fca434af04377a2cbf99e96abc42948bf738c06e756" exitCode=0
Aug 01 00:32:56.722703 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:32:56.722353    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" event={"ID":"41ca9cc3-b80d-4f0b-b941-a93c18813f86","Type":"ContainerDied","Data":"dc9beb25f9cd838dd7021fca434af04377a2cbf99e96abc42948bf738c06e756"}
Aug 01 00:33:08.786959 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:33:08.786925    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" event={"ID":"41ca9cc3-b80d-4f0b-b941-a93c18813f86","Type":"ContainerStarted","Data":"8c9d8fc7583fcfc3d72ca86389f4b1c9b89bcbf2a35582b18fd46aae6aa8527e"}
Aug 01 00:33:08.787446 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:33:08.787009    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" event={"ID":"41ca9cc3-b80d-4f0b-b941-a93c18813f86","Type":"ContainerStarted","Data":"777888a4f20d9a3a16f4ba93517c36c15d9d333b44fdbe3386342e8d568de963"}
Aug 01 00:33:08.787446 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:33:08.787230    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:33:08.806210 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:33:08.806167    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" podStartSLOduration=2.36993431 podStartE2EDuration="18.806154265s" totalImagesPullingTime="11.542068719s" totalInitContainerRuntime="4.894151236s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:33:08.804848782 +0000 UTC m=+2823.678492407" watchObservedRunningTime="2026-08-01 00:33:08.806154265 +0000 UTC m=+2823.679797875"
Aug 01 00:33:09.791249 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:33:09.791221    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:33:09.792637 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:33:09.792616    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Aug 01 00:33:10.795427 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:33:10.795385    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Aug 01 00:33:15.799884 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:33:15.799856    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:33:15.800384 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:33:15.800358    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Aug 01 00:33:25.801177 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:33:25.801138    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Aug 01 00:33:35.801129 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:33:35.801090    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Aug 01 00:33:45.800980 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:33:45.800935    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Aug 01 00:33:55.801728 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:33:55.801700    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:34:02.247948 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:02.247874    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"]
Aug 01 00:34:02.248451 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:02.248131    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="kserve-container" containerID="cri-o://777888a4f20d9a3a16f4ba93517c36c15d9d333b44fdbe3386342e8d568de963" gracePeriod=30
Aug 01 00:34:02.248451 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:02.248188    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="kube-rbac-proxy" containerID="cri-o://8c9d8fc7583fcfc3d72ca86389f4b1c9b89bcbf2a35582b18fd46aae6aa8527e" gracePeriod=30
Aug 01 00:34:03.017880 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:03.017848    2584 generic.go:356] "Generic (PLEG): container finished" podID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerID="8c9d8fc7583fcfc3d72ca86389f4b1c9b89bcbf2a35582b18fd46aae6aa8527e" exitCode=2
Aug 01 00:34:03.018075 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:03.017934    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" event={"ID":"41ca9cc3-b80d-4f0b-b941-a93c18813f86","Type":"ContainerDied","Data":"8c9d8fc7583fcfc3d72ca86389f4b1c9b89bcbf2a35582b18fd46aae6aa8527e"}
Aug 01 00:34:04.990759 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:04.990730    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:34:05.028850 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.028819    2584 generic.go:356] "Generic (PLEG): container finished" podID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerID="777888a4f20d9a3a16f4ba93517c36c15d9d333b44fdbe3386342e8d568de963" exitCode=0
Aug 01 00:34:05.029031 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.028872    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"
Aug 01 00:34:05.029031 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.028882    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" event={"ID":"41ca9cc3-b80d-4f0b-b941-a93c18813f86","Type":"ContainerDied","Data":"777888a4f20d9a3a16f4ba93517c36c15d9d333b44fdbe3386342e8d568de963"}
Aug 01 00:34:05.029031 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.028935    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59" event={"ID":"41ca9cc3-b80d-4f0b-b941-a93c18813f86","Type":"ContainerDied","Data":"2782029c3b793af55a2c29e6fad52c2babf9ddf6db19ac682bbedacaba8404ff"}
Aug 01 00:34:05.029031 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.028982    2584 scope.go:117] "RemoveContainer" containerID="8c9d8fc7583fcfc3d72ca86389f4b1c9b89bcbf2a35582b18fd46aae6aa8527e"
Aug 01 00:34:05.036684 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.036666    2584 scope.go:117] "RemoveContainer" containerID="777888a4f20d9a3a16f4ba93517c36c15d9d333b44fdbe3386342e8d568de963"
Aug 01 00:34:05.043583 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.043567    2584 scope.go:117] "RemoveContainer" containerID="dc9beb25f9cd838dd7021fca434af04377a2cbf99e96abc42948bf738c06e756"
Aug 01 00:34:05.050546 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.050531    2584 scope.go:117] "RemoveContainer" containerID="8c9d8fc7583fcfc3d72ca86389f4b1c9b89bcbf2a35582b18fd46aae6aa8527e"
Aug 01 00:34:05.050805 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:34:05.050781    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9d8fc7583fcfc3d72ca86389f4b1c9b89bcbf2a35582b18fd46aae6aa8527e\": container with ID starting with 8c9d8fc7583fcfc3d72ca86389f4b1c9b89bcbf2a35582b18fd46aae6aa8527e not found: ID does not exist" containerID="8c9d8fc7583fcfc3d72ca86389f4b1c9b89bcbf2a35582b18fd46aae6aa8527e"
Aug 01 00:34:05.050899 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.050813    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9d8fc7583fcfc3d72ca86389f4b1c9b89bcbf2a35582b18fd46aae6aa8527e"} err="failed to get container status \"8c9d8fc7583fcfc3d72ca86389f4b1c9b89bcbf2a35582b18fd46aae6aa8527e\": rpc error: code = NotFound desc = could not find container \"8c9d8fc7583fcfc3d72ca86389f4b1c9b89bcbf2a35582b18fd46aae6aa8527e\": container with ID starting with 8c9d8fc7583fcfc3d72ca86389f4b1c9b89bcbf2a35582b18fd46aae6aa8527e not found: ID does not exist"
Aug 01 00:34:05.050899 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.050831    2584 scope.go:117] "RemoveContainer" containerID="777888a4f20d9a3a16f4ba93517c36c15d9d333b44fdbe3386342e8d568de963"
Aug 01 00:34:05.051133 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:34:05.051100    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777888a4f20d9a3a16f4ba93517c36c15d9d333b44fdbe3386342e8d568de963\": container with ID starting with 777888a4f20d9a3a16f4ba93517c36c15d9d333b44fdbe3386342e8d568de963 not found: ID does not exist" containerID="777888a4f20d9a3a16f4ba93517c36c15d9d333b44fdbe3386342e8d568de963"
Aug 01 00:34:05.051213 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.051133    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777888a4f20d9a3a16f4ba93517c36c15d9d333b44fdbe3386342e8d568de963"} err="failed to get container status \"777888a4f20d9a3a16f4ba93517c36c15d9d333b44fdbe3386342e8d568de963\": rpc error: code = NotFound desc = could not find container \"777888a4f20d9a3a16f4ba93517c36c15d9d333b44fdbe3386342e8d568de963\": container with ID starting with 777888a4f20d9a3a16f4ba93517c36c15d9d333b44fdbe3386342e8d568de963 not found: ID does not exist"
Aug 01 00:34:05.051213 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.051149    2584 scope.go:117] "RemoveContainer" containerID="dc9beb25f9cd838dd7021fca434af04377a2cbf99e96abc42948bf738c06e756"
Aug 01 00:34:05.051391 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:34:05.051354    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9beb25f9cd838dd7021fca434af04377a2cbf99e96abc42948bf738c06e756\": container with ID starting with dc9beb25f9cd838dd7021fca434af04377a2cbf99e96abc42948bf738c06e756 not found: ID does not exist" containerID="dc9beb25f9cd838dd7021fca434af04377a2cbf99e96abc42948bf738c06e756"
Aug 01 00:34:05.051391 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.051377    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9beb25f9cd838dd7021fca434af04377a2cbf99e96abc42948bf738c06e756"} err="failed to get container status \"dc9beb25f9cd838dd7021fca434af04377a2cbf99e96abc42948bf738c06e756\": rpc error: code = NotFound desc = could not find container \"dc9beb25f9cd838dd7021fca434af04377a2cbf99e96abc42948bf738c06e756\": container with ID starting with dc9beb25f9cd838dd7021fca434af04377a2cbf99e96abc42948bf738c06e756 not found: ID does not exist"
Aug 01 00:34:05.061682 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.061660    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b8n5\" (UniqueName: \"kubernetes.io/projected/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kube-api-access-2b8n5\") pod \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") "
Aug 01 00:34:05.061767 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.061695    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41ca9cc3-b80d-4f0b-b941-a93c18813f86-cabundle-cert\") pod \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") "
Aug 01 00:34:05.061767 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.061740    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kserve-provision-location\") pod \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") "
Aug 01 00:34:05.061854 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.061836    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kube-rbac-proxy-sar-config\") pod \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") "
Aug 01 00:34:05.061902 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.061882    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ca9cc3-b80d-4f0b-b941-a93c18813f86-proxy-tls\") pod \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\" (UID: \"41ca9cc3-b80d-4f0b-b941-a93c18813f86\") "
Aug 01 00:34:05.062154 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.062131    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ca9cc3-b80d-4f0b-b941-a93c18813f86-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41ca9cc3-b80d-4f0b-b941-a93c18813f86" (UID: "41ca9cc3-b80d-4f0b-b941-a93c18813f86"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:34:05.062246 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.062205    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41ca9cc3-b80d-4f0b-b941-a93c18813f86" (UID: "41ca9cc3-b80d-4f0b-b941-a93c18813f86"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:34:05.063680 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.063652    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kube-api-access-2b8n5" (OuterVolumeSpecName: "kube-api-access-2b8n5") pod "41ca9cc3-b80d-4f0b-b941-a93c18813f86" (UID: "41ca9cc3-b80d-4f0b-b941-a93c18813f86"). InnerVolumeSpecName "kube-api-access-2b8n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:34:05.063826 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.063800    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ca9cc3-b80d-4f0b-b941-a93c18813f86-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41ca9cc3-b80d-4f0b-b941-a93c18813f86" (UID: "41ca9cc3-b80d-4f0b-b941-a93c18813f86"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:34:05.071949 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.071927    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41ca9cc3-b80d-4f0b-b941-a93c18813f86" (UID: "41ca9cc3-b80d-4f0b-b941-a93c18813f86"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:34:05.163351 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.163272    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:34:05.163351 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.163297    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:34:05.163351 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.163307    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ca9cc3-b80d-4f0b-b941-a93c18813f86-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:34:05.163351 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.163317    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b8n5\" (UniqueName: \"kubernetes.io/projected/41ca9cc3-b80d-4f0b-b941-a93c18813f86-kube-api-access-2b8n5\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:34:05.163351 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.163325    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41ca9cc3-b80d-4f0b-b941-a93c18813f86-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:34:05.350802 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.350774    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"]
Aug 01 00:34:05.354303 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.354281    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p4g59"]
Aug 01 00:34:05.570302 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:05.570273    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" path="/var/lib/kubelet/pods/41ca9cc3-b80d-4f0b-b941-a93c18813f86/volumes"
Aug 01 00:34:07.511631 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.511599    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"]
Aug 01 00:34:07.512056 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.512036    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="storage-initializer"
Aug 01 00:34:07.512115 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.512060    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="storage-initializer"
Aug 01 00:34:07.512115 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.512082    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="kube-rbac-proxy"
Aug 01 00:34:07.512115 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.512091    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="kube-rbac-proxy"
Aug 01 00:34:07.512115 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.512109    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="kserve-container"
Aug 01 00:34:07.512313 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.512119    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="kserve-container"
Aug 01 00:34:07.512313 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.512201    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="kserve-container"
Aug 01 00:34:07.512313 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.512217    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ca9cc3-b80d-4f0b-b941-a93c18813f86" containerName="kube-rbac-proxy"
Aug 01 00:34:07.515364 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.515347    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.517990 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.517958    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Aug 01 00:34:07.518112 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.518089    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:34:07.518313 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.518298    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Aug 01 00:34:07.518415 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.518332    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:34:07.519235 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.519212    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:34:07.519323 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.519238    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:34:07.519323 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.519302    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:34:07.525094 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.525071    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"]
Aug 01 00:34:07.580941 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.580909    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b3d1ac5-7789-4703-ba36-e9126bb491ee-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.580941 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.580944    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.581230 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.581030    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b3d1ac5-7789-4703-ba36-e9126bb491ee-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.581230 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.581094    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.581230 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.581125    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55dzr\" (UniqueName: \"kubernetes.io/projected/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kube-api-access-55dzr\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.682312 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.682274    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b3d1ac5-7789-4703-ba36-e9126bb491ee-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.682495 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.682319    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.682495 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.682352    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b3d1ac5-7789-4703-ba36-e9126bb491ee-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.682495 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.682384    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.682495 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.682414    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55dzr\" (UniqueName: \"kubernetes.io/projected/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kube-api-access-55dzr\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.682824 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.682798    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.683017 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.682995    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b3d1ac5-7789-4703-ba36-e9126bb491ee-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.683103 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.683078    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.684998 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.684977    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b3d1ac5-7789-4703-ba36-e9126bb491ee-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.690277 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.690257    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55dzr\" (UniqueName: \"kubernetes.io/projected/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kube-api-access-55dzr\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.826140 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.826108    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:07.947315 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:07.947291    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"]
Aug 01 00:34:07.949244 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:34:07.949215    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3d1ac5_7789_4703_ba36_e9126bb491ee.slice/crio-26eb3795d9c48f3211d8fb008eb46dd52f6a7c070d4ae0510acb1a188ab27349 WatchSource:0}: Error finding container 26eb3795d9c48f3211d8fb008eb46dd52f6a7c070d4ae0510acb1a188ab27349: Status 404 returned error can't find the container with id 26eb3795d9c48f3211d8fb008eb46dd52f6a7c070d4ae0510acb1a188ab27349
Aug 01 00:34:08.043675 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:08.043633    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" event={"ID":"2b3d1ac5-7789-4703-ba36-e9126bb491ee","Type":"ContainerStarted","Data":"bc659ea85512287b301f2bb027554898ea83ff7f8d00dab74cf4a7abf154fb56"}
Aug 01 00:34:08.043795 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:08.043714    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" event={"ID":"2b3d1ac5-7789-4703-ba36-e9126bb491ee","Type":"ContainerStarted","Data":"26eb3795d9c48f3211d8fb008eb46dd52f6a7c070d4ae0510acb1a188ab27349"}
Aug 01 00:34:13.068141 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:13.068097    2584 generic.go:356] "Generic (PLEG): container finished" podID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerID="bc659ea85512287b301f2bb027554898ea83ff7f8d00dab74cf4a7abf154fb56" exitCode=0
Aug 01 00:34:13.068556 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:13.068180    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" event={"ID":"2b3d1ac5-7789-4703-ba36-e9126bb491ee","Type":"ContainerDied","Data":"bc659ea85512287b301f2bb027554898ea83ff7f8d00dab74cf4a7abf154fb56"}
Aug 01 00:34:14.074173 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:14.074138    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" event={"ID":"2b3d1ac5-7789-4703-ba36-e9126bb491ee","Type":"ContainerStarted","Data":"556ce57b0fecd946a68eae2d23716298342858edd34a38550122810d27eb75e5"}
Aug 01 00:34:14.074636 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:14.074230    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" event={"ID":"2b3d1ac5-7789-4703-ba36-e9126bb491ee","Type":"ContainerStarted","Data":"84b93bb46002c5d2f1ff751761ea03c287cdfdb5f48938b34916aa5b48cba896"}
Aug 01 00:34:14.074636 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:14.074423    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:14.091919 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:14.091876    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" podStartSLOduration=2.774931105 podStartE2EDuration="7.091859467s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.316928362s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:14.091640448 +0000 UTC m=+2888.965284059" watchObservedRunningTime="2026-08-01 00:34:14.091859467 +0000 UTC m=+2888.965503079"
Aug 01 00:34:15.078035 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:15.077997    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:15.079336 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:15.079310    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Aug 01 00:34:16.082514 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:16.082474    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Aug 01 00:34:21.087030 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:21.086997    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:34:21.087596 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:21.087454    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Aug 01 00:34:31.087734 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:31.087699    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Aug 01 00:34:41.088173 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:41.088132    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Aug 01 00:34:51.088380 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:34:51.088336    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Aug 01 00:35:01.087943 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:01.087914    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:35:09.371714 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:09.371684    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"]
Aug 01 00:35:09.372159 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:09.371908    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kserve-container" containerID="cri-o://84b93bb46002c5d2f1ff751761ea03c287cdfdb5f48938b34916aa5b48cba896" gracePeriod=30
Aug 01 00:35:09.372159 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:09.371954    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kube-rbac-proxy" containerID="cri-o://556ce57b0fecd946a68eae2d23716298342858edd34a38550122810d27eb75e5" gracePeriod=30
Aug 01 00:35:10.312879 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:10.312842    2584 generic.go:356] "Generic (PLEG): container finished" podID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerID="556ce57b0fecd946a68eae2d23716298342858edd34a38550122810d27eb75e5" exitCode=2
Aug 01 00:35:10.313068 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:10.312913    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" event={"ID":"2b3d1ac5-7789-4703-ba36-e9126bb491ee","Type":"ContainerDied","Data":"556ce57b0fecd946a68eae2d23716298342858edd34a38550122810d27eb75e5"}
Aug 01 00:35:11.082826 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:11.082785    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Aug 01 00:35:11.088191 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:11.088152    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Aug 01 00:35:12.119447 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.119417    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:35:12.197032 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.196943    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b3d1ac5-7789-4703-ba36-e9126bb491ee-cabundle-cert\") pod \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") "
Aug 01 00:35:12.197032 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.197023    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b3d1ac5-7789-4703-ba36-e9126bb491ee-proxy-tls\") pod \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") "
Aug 01 00:35:12.197263 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.197102    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kube-rbac-proxy-sar-config\") pod \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") "
Aug 01 00:35:12.197263 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.197150    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55dzr\" (UniqueName: \"kubernetes.io/projected/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kube-api-access-55dzr\") pod \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") "
Aug 01 00:35:12.197263 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.197188    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kserve-provision-location\") pod \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\" (UID: \"2b3d1ac5-7789-4703-ba36-e9126bb491ee\") "
Aug 01 00:35:12.197425 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.197359    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3d1ac5-7789-4703-ba36-e9126bb491ee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2b3d1ac5-7789-4703-ba36-e9126bb491ee" (UID: "2b3d1ac5-7789-4703-ba36-e9126bb491ee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:35:12.197566 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.197541    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2b3d1ac5-7789-4703-ba36-e9126bb491ee" (UID: "2b3d1ac5-7789-4703-ba36-e9126bb491ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:35:12.199199 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.199173    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kube-api-access-55dzr" (OuterVolumeSpecName: "kube-api-access-55dzr") pod "2b3d1ac5-7789-4703-ba36-e9126bb491ee" (UID: "2b3d1ac5-7789-4703-ba36-e9126bb491ee"). InnerVolumeSpecName "kube-api-access-55dzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:35:12.199287 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.199200    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b3d1ac5-7789-4703-ba36-e9126bb491ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b3d1ac5-7789-4703-ba36-e9126bb491ee" (UID: "2b3d1ac5-7789-4703-ba36-e9126bb491ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:35:12.210196 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.210169    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b3d1ac5-7789-4703-ba36-e9126bb491ee" (UID: "2b3d1ac5-7789-4703-ba36-e9126bb491ee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:35:12.297768 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.297735    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b3d1ac5-7789-4703-ba36-e9126bb491ee-cabundle-cert\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:35:12.297768 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.297763    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b3d1ac5-7789-4703-ba36-e9126bb491ee-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:35:12.297768 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.297774    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:35:12.298029 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.297783    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55dzr\" (UniqueName: \"kubernetes.io/projected/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kube-api-access-55dzr\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:35:12.298029 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.297792    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b3d1ac5-7789-4703-ba36-e9126bb491ee-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:35:12.322939 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.322905    2584 generic.go:356] "Generic (PLEG): container finished" podID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerID="84b93bb46002c5d2f1ff751761ea03c287cdfdb5f48938b34916aa5b48cba896" exitCode=0
Aug 01 00:35:12.323106 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.322995    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"
Aug 01 00:35:12.323106 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.323008    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" event={"ID":"2b3d1ac5-7789-4703-ba36-e9126bb491ee","Type":"ContainerDied","Data":"84b93bb46002c5d2f1ff751761ea03c287cdfdb5f48938b34916aa5b48cba896"}
Aug 01 00:35:12.323106 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.323072    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft" event={"ID":"2b3d1ac5-7789-4703-ba36-e9126bb491ee","Type":"ContainerDied","Data":"26eb3795d9c48f3211d8fb008eb46dd52f6a7c070d4ae0510acb1a188ab27349"}
Aug 01 00:35:12.323106 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.323103    2584 scope.go:117] "RemoveContainer" containerID="556ce57b0fecd946a68eae2d23716298342858edd34a38550122810d27eb75e5"
Aug 01 00:35:12.331704 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.331686    2584 scope.go:117] "RemoveContainer" containerID="84b93bb46002c5d2f1ff751761ea03c287cdfdb5f48938b34916aa5b48cba896"
Aug 01 00:35:12.338954 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.338936    2584 scope.go:117] "RemoveContainer" containerID="bc659ea85512287b301f2bb027554898ea83ff7f8d00dab74cf4a7abf154fb56"
Aug 01 00:35:12.345383 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.345362    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"]
Aug 01 00:35:12.346289 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.346264    2584 scope.go:117] "RemoveContainer" containerID="556ce57b0fecd946a68eae2d23716298342858edd34a38550122810d27eb75e5"
Aug 01 00:35:12.346589 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:35:12.346568    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556ce57b0fecd946a68eae2d23716298342858edd34a38550122810d27eb75e5\": container with ID starting with 556ce57b0fecd946a68eae2d23716298342858edd34a38550122810d27eb75e5 not found: ID does not exist" containerID="556ce57b0fecd946a68eae2d23716298342858edd34a38550122810d27eb75e5"
Aug 01 00:35:12.346683 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.346601    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556ce57b0fecd946a68eae2d23716298342858edd34a38550122810d27eb75e5"} err="failed to get container status \"556ce57b0fecd946a68eae2d23716298342858edd34a38550122810d27eb75e5\": rpc error: code = NotFound desc = could not find container \"556ce57b0fecd946a68eae2d23716298342858edd34a38550122810d27eb75e5\": container with ID starting with 556ce57b0fecd946a68eae2d23716298342858edd34a38550122810d27eb75e5 not found: ID does not exist"
Aug 01 00:35:12.346683 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.346626    2584 scope.go:117] "RemoveContainer" containerID="84b93bb46002c5d2f1ff751761ea03c287cdfdb5f48938b34916aa5b48cba896"
Aug 01 00:35:12.347015 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:35:12.346941    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b93bb46002c5d2f1ff751761ea03c287cdfdb5f48938b34916aa5b48cba896\": container with ID starting with 84b93bb46002c5d2f1ff751761ea03c287cdfdb5f48938b34916aa5b48cba896 not found: ID does not exist" containerID="84b93bb46002c5d2f1ff751761ea03c287cdfdb5f48938b34916aa5b48cba896"
Aug 01 00:35:12.347015 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.346986    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b93bb46002c5d2f1ff751761ea03c287cdfdb5f48938b34916aa5b48cba896"} err="failed to get container status \"84b93bb46002c5d2f1ff751761ea03c287cdfdb5f48938b34916aa5b48cba896\": rpc error: code = NotFound desc = could not find container \"84b93bb46002c5d2f1ff751761ea03c287cdfdb5f48938b34916aa5b48cba896\": container with ID starting with 84b93bb46002c5d2f1ff751761ea03c287cdfdb5f48938b34916aa5b48cba896 not found: ID does not exist"
Aug 01 00:35:12.347015 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.347005    2584 scope.go:117] "RemoveContainer" containerID="bc659ea85512287b301f2bb027554898ea83ff7f8d00dab74cf4a7abf154fb56"
Aug 01 00:35:12.347378 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:35:12.347315    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc659ea85512287b301f2bb027554898ea83ff7f8d00dab74cf4a7abf154fb56\": container with ID starting with bc659ea85512287b301f2bb027554898ea83ff7f8d00dab74cf4a7abf154fb56 not found: ID does not exist" containerID="bc659ea85512287b301f2bb027554898ea83ff7f8d00dab74cf4a7abf154fb56"
Aug 01 00:35:12.347378 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.347342    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc659ea85512287b301f2bb027554898ea83ff7f8d00dab74cf4a7abf154fb56"} err="failed to get container status \"bc659ea85512287b301f2bb027554898ea83ff7f8d00dab74cf4a7abf154fb56\": rpc error: code = NotFound desc = could not find container \"bc659ea85512287b301f2bb027554898ea83ff7f8d00dab74cf4a7abf154fb56\": container with ID starting with bc659ea85512287b301f2bb027554898ea83ff7f8d00dab74cf4a7abf154fb56 not found: ID does not exist"
Aug 01 00:35:12.348923 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:12.348905    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-rmpft"]
Aug 01 00:35:13.569832 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:13.569798    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" path="/var/lib/kubelet/pods/2b3d1ac5-7789-4703-ba36-e9126bb491ee/volumes"
Aug 01 00:35:50.732194 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.732157    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"]
Aug 01 00:35:50.732619 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.732450    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kube-rbac-proxy"
Aug 01 00:35:50.732619 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.732462    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kube-rbac-proxy"
Aug 01 00:35:50.732619 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.732474    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kserve-container"
Aug 01 00:35:50.732619 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.732479    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kserve-container"
Aug 01 00:35:50.732619 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.732490    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="storage-initializer"
Aug 01 00:35:50.732619 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.732498    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="storage-initializer"
Aug 01 00:35:50.732619 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.732544    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kserve-container"
Aug 01 00:35:50.732619 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.732550    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b3d1ac5-7789-4703-ba36-e9126bb491ee" containerName="kube-rbac-proxy"
Aug 01 00:35:50.735624 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.735606    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:35:50.740192 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.740172    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Aug 01 00:35:50.740462 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.740445    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Aug 01 00:35:50.740873 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.740856    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:35:50.740978 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.740861    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Aug 01 00:35:50.740978 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.740927    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:35:50.740978 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.740872    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:35:50.746637 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.746617    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"]
Aug 01 00:35:50.813466 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.813432    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7da643a3-4515-422c-8640-ef146a6731b8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:35:50.813644 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.813482    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvd5t\" (UniqueName: \"kubernetes.io/projected/7da643a3-4515-422c-8640-ef146a6731b8-kube-api-access-jvd5t\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:35:50.813644 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.813515    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7da643a3-4515-422c-8640-ef146a6731b8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:35:50.813644 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.813545    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7da643a3-4515-422c-8640-ef146a6731b8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:35:50.914573 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.914534    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7da643a3-4515-422c-8640-ef146a6731b8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:35:50.914750 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.914588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvd5t\" (UniqueName: \"kubernetes.io/projected/7da643a3-4515-422c-8640-ef146a6731b8-kube-api-access-jvd5t\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:35:50.914750 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.914620    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7da643a3-4515-422c-8640-ef146a6731b8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:35:50.914750 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.914668    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7da643a3-4515-422c-8640-ef146a6731b8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:35:50.914890 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:35:50.914757    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Aug 01 00:35:50.914890 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:35:50.914822    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da643a3-4515-422c-8640-ef146a6731b8-proxy-tls podName:7da643a3-4515-422c-8640-ef146a6731b8 nodeName:}" failed. No retries permitted until 2026-08-01 00:35:51.414802487 +0000 UTC m=+2986.288446089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7da643a3-4515-422c-8640-ef146a6731b8-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l" (UID: "7da643a3-4515-422c-8640-ef146a6731b8") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Aug 01 00:35:50.915070 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.915051    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7da643a3-4515-422c-8640-ef146a6731b8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:35:50.915291 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.915273    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7da643a3-4515-422c-8640-ef146a6731b8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:35:50.923510 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:50.923485    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvd5t\" (UniqueName: \"kubernetes.io/projected/7da643a3-4515-422c-8640-ef146a6731b8-kube-api-access-jvd5t\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:35:51.418189 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:51.418154    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7da643a3-4515-422c-8640-ef146a6731b8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:35:51.420553 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:51.420528    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7da643a3-4515-422c-8640-ef146a6731b8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:35:51.646684 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:51.646642    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:35:51.775810 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:51.775786    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"]
Aug 01 00:35:51.777697 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:35:51.777673    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da643a3_4515_422c_8640_ef146a6731b8.slice/crio-413896a9f6f9cbf3e6397f0cbc30e612d49ea56cba8e8525d80ce672681d7025 WatchSource:0}: Error finding container 413896a9f6f9cbf3e6397f0cbc30e612d49ea56cba8e8525d80ce672681d7025: Status 404 returned error can't find the container with id 413896a9f6f9cbf3e6397f0cbc30e612d49ea56cba8e8525d80ce672681d7025
Aug 01 00:35:52.491878 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:52.491841    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l" event={"ID":"7da643a3-4515-422c-8640-ef146a6731b8","Type":"ContainerStarted","Data":"3f52942918c179bf45bc2f4539043d61cf1cce7f454b7c589dd5eacd5807dd4f"}
Aug 01 00:35:52.492080 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:52.491913    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l" event={"ID":"7da643a3-4515-422c-8640-ef146a6731b8","Type":"ContainerStarted","Data":"413896a9f6f9cbf3e6397f0cbc30e612d49ea56cba8e8525d80ce672681d7025"}
Aug 01 00:35:54.502529 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:54.502495    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l_7da643a3-4515-422c-8640-ef146a6731b8/storage-initializer/0.log"
Aug 01 00:35:54.503028 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:54.502540    2584 generic.go:356] "Generic (PLEG): container finished" podID="7da643a3-4515-422c-8640-ef146a6731b8" containerID="3f52942918c179bf45bc2f4539043d61cf1cce7f454b7c589dd5eacd5807dd4f" exitCode=1
Aug 01 00:35:54.503028 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:54.502602    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l" event={"ID":"7da643a3-4515-422c-8640-ef146a6731b8","Type":"ContainerDied","Data":"3f52942918c179bf45bc2f4539043d61cf1cce7f454b7c589dd5eacd5807dd4f"}
Aug 01 00:35:55.509326 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:55.509300    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l_7da643a3-4515-422c-8640-ef146a6731b8/storage-initializer/0.log"
Aug 01 00:35:55.509766 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:35:55.509410    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l" event={"ID":"7da643a3-4515-422c-8640-ef146a6731b8","Type":"ContainerStarted","Data":"ffd6ae48a022875d93bfbed0b0de93a43aadc2b12b6ca9e555382da9c7d2bf5a"}
Aug 01 00:36:00.531805 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:00.531776    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l_7da643a3-4515-422c-8640-ef146a6731b8/storage-initializer/1.log"
Aug 01 00:36:00.532237 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:00.532162    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l_7da643a3-4515-422c-8640-ef146a6731b8/storage-initializer/0.log"
Aug 01 00:36:00.532237 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:00.532198    2584 generic.go:356] "Generic (PLEG): container finished" podID="7da643a3-4515-422c-8640-ef146a6731b8" containerID="ffd6ae48a022875d93bfbed0b0de93a43aadc2b12b6ca9e555382da9c7d2bf5a" exitCode=1
Aug 01 00:36:00.532324 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:00.532273    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l" event={"ID":"7da643a3-4515-422c-8640-ef146a6731b8","Type":"ContainerDied","Data":"ffd6ae48a022875d93bfbed0b0de93a43aadc2b12b6ca9e555382da9c7d2bf5a"}
Aug 01 00:36:00.532360 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:00.532339    2584 scope.go:117] "RemoveContainer" containerID="3f52942918c179bf45bc2f4539043d61cf1cce7f454b7c589dd5eacd5807dd4f"
Aug 01 00:36:00.532665 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:00.532646    2584 scope.go:117] "RemoveContainer" containerID="3f52942918c179bf45bc2f4539043d61cf1cce7f454b7c589dd5eacd5807dd4f"
Aug 01 00:36:00.542857 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:36:00.542833    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l_kserve-ci-e2e-test(7da643a3-4515-422c-8640-ef146a6731b8)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l" podUID="7da643a3-4515-422c-8640-ef146a6731b8"
Aug 01 00:36:00.542950 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:36:00.542928    2584 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l_kserve-ci-e2e-test_7da643a3-4515-422c-8640-ef146a6731b8_0 in pod sandbox 413896a9f6f9cbf3e6397f0cbc30e612d49ea56cba8e8525d80ce672681d7025 from index: no such id: '3f52942918c179bf45bc2f4539043d61cf1cce7f454b7c589dd5eacd5807dd4f'" containerID="3f52942918c179bf45bc2f4539043d61cf1cce7f454b7c589dd5eacd5807dd4f"
Aug 01 00:36:00.543021 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:00.542973    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f52942918c179bf45bc2f4539043d61cf1cce7f454b7c589dd5eacd5807dd4f"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l_kserve-ci-e2e-test_7da643a3-4515-422c-8640-ef146a6731b8_0 in pod sandbox 413896a9f6f9cbf3e6397f0cbc30e612d49ea56cba8e8525d80ce672681d7025 from index: no such id: '3f52942918c179bf45bc2f4539043d61cf1cce7f454b7c589dd5eacd5807dd4f'"
Aug 01 00:36:00.740605 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:00.740574    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"]
Aug 01 00:36:01.538134 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:01.538105    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l_7da643a3-4515-422c-8640-ef146a6731b8/storage-initializer/1.log"
Aug 01 00:36:01.659176 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:01.659152    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l_7da643a3-4515-422c-8640-ef146a6731b8/storage-initializer/1.log"
Aug 01 00:36:01.662655 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:01.662637    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:36:01.807748 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:01.807705    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7da643a3-4515-422c-8640-ef146a6731b8-kserve-provision-location\") pod \"7da643a3-4515-422c-8640-ef146a6731b8\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") "
Aug 01 00:36:01.807917 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:01.807760    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7da643a3-4515-422c-8640-ef146a6731b8-kube-rbac-proxy-sar-config\") pod \"7da643a3-4515-422c-8640-ef146a6731b8\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") "
Aug 01 00:36:01.807917 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:01.807795    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvd5t\" (UniqueName: \"kubernetes.io/projected/7da643a3-4515-422c-8640-ef146a6731b8-kube-api-access-jvd5t\") pod \"7da643a3-4515-422c-8640-ef146a6731b8\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") "
Aug 01 00:36:01.807917 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:01.807814    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7da643a3-4515-422c-8640-ef146a6731b8-proxy-tls\") pod \"7da643a3-4515-422c-8640-ef146a6731b8\" (UID: \"7da643a3-4515-422c-8640-ef146a6731b8\") "
Aug 01 00:36:01.808121 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:01.808017    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da643a3-4515-422c-8640-ef146a6731b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7da643a3-4515-422c-8640-ef146a6731b8" (UID: "7da643a3-4515-422c-8640-ef146a6731b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:36:01.808191 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:01.808118    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7da643a3-4515-422c-8640-ef146a6731b8-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:36:01.808257 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:01.808185    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7da643a3-4515-422c-8640-ef146a6731b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7da643a3-4515-422c-8640-ef146a6731b8" (UID: "7da643a3-4515-422c-8640-ef146a6731b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:36:01.810113 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:01.810090    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7da643a3-4515-422c-8640-ef146a6731b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7da643a3-4515-422c-8640-ef146a6731b8" (UID: "7da643a3-4515-422c-8640-ef146a6731b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:36:01.810335 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:01.810313    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da643a3-4515-422c-8640-ef146a6731b8-kube-api-access-jvd5t" (OuterVolumeSpecName: "kube-api-access-jvd5t") pod "7da643a3-4515-422c-8640-ef146a6731b8" (UID: "7da643a3-4515-422c-8640-ef146a6731b8"). InnerVolumeSpecName "kube-api-access-jvd5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:36:01.909138 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:01.909103    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7da643a3-4515-422c-8640-ef146a6731b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:36:01.909138 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:01.909135    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvd5t\" (UniqueName: \"kubernetes.io/projected/7da643a3-4515-422c-8640-ef146a6731b8-kube-api-access-jvd5t\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:36:01.909340 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:01.909147    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7da643a3-4515-422c-8640-ef146a6731b8-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:36:02.543256 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:02.543227    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l_7da643a3-4515-422c-8640-ef146a6731b8/storage-initializer/1.log"
Aug 01 00:36:02.543656 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:02.543291    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l" event={"ID":"7da643a3-4515-422c-8640-ef146a6731b8","Type":"ContainerDied","Data":"413896a9f6f9cbf3e6397f0cbc30e612d49ea56cba8e8525d80ce672681d7025"}
Aug 01 00:36:02.543656 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:02.543337    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"
Aug 01 00:36:02.543656 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:02.543345    2584 scope.go:117] "RemoveContainer" containerID="ffd6ae48a022875d93bfbed0b0de93a43aadc2b12b6ca9e555382da9c7d2bf5a"
Aug 01 00:36:02.579929 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:02.579896    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"]
Aug 01 00:36:02.585958 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:02.585913    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-596f78f4cf-klc4l"]
Aug 01 00:36:03.570778 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:03.570745    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7da643a3-4515-422c-8640-ef146a6731b8" path="/var/lib/kubelet/pods/7da643a3-4515-422c-8640-ef146a6731b8/volumes"
Aug 01 00:36:05.729607 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:05.729577    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:36:05.736148 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:36:05.736124    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:37:23.006393 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.006314    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"]
Aug 01 00:37:23.006842 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.006644    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7da643a3-4515-422c-8640-ef146a6731b8" containerName="storage-initializer"
Aug 01 00:37:23.006842 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.006656    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7da643a3-4515-422c-8640-ef146a6731b8" containerName="storage-initializer"
Aug 01 00:37:23.006842 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.006671    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7da643a3-4515-422c-8640-ef146a6731b8" containerName="storage-initializer"
Aug 01 00:37:23.006842 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.006677    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7da643a3-4515-422c-8640-ef146a6731b8" containerName="storage-initializer"
Aug 01 00:37:23.006842 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.006736    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7da643a3-4515-422c-8640-ef146a6731b8" containerName="storage-initializer"
Aug 01 00:37:23.006842 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.006838    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7da643a3-4515-422c-8640-ef146a6731b8" containerName="storage-initializer"
Aug 01 00:37:23.009928 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.009913    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:23.019636 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.019606    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Aug 01 00:37:23.019759 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.019649    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Aug 01 00:37:23.019759 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.019690    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:37:23.019759 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.019697    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:37:23.019759 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.019690    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:37:23.019759 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.019651    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Aug 01 00:37:23.033433 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.033410    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"]
Aug 01 00:37:23.042361 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.042303    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/801862f1-32f0-4e77-84bf-ad4895e9f6ac-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:23.042504 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.042403    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:23.042504 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.042434    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tphvs\" (UniqueName: \"kubernetes.io/projected/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kube-api-access-tphvs\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:23.042504 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.042470    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:23.143668 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.143634    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/801862f1-32f0-4e77-84bf-ad4895e9f6ac-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:23.143868 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.143700    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:23.143868 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.143805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tphvs\" (UniqueName: \"kubernetes.io/projected/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kube-api-access-tphvs\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:23.143868 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.143852    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:23.144168 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.144141    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:23.144531 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.144511    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:23.146270 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.146252    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/801862f1-32f0-4e77-84bf-ad4895e9f6ac-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:23.151345 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.151322    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tphvs\" (UniqueName: \"kubernetes.io/projected/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kube-api-access-tphvs\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:23.319627 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.319596    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:23.446088 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.446055    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"]
Aug 01 00:37:23.447483 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:37:23.447454    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801862f1_32f0_4e77_84bf_ad4895e9f6ac.slice/crio-4f9c15f79a26a70a7a89c623413533f690d0c2c92cf50f0c3863edb38876f43b WatchSource:0}: Error finding container 4f9c15f79a26a70a7a89c623413533f690d0c2c92cf50f0c3863edb38876f43b: Status 404 returned error can't find the container with id 4f9c15f79a26a70a7a89c623413533f690d0c2c92cf50f0c3863edb38876f43b
Aug 01 00:37:23.894293 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.894258    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf" event={"ID":"801862f1-32f0-4e77-84bf-ad4895e9f6ac","Type":"ContainerStarted","Data":"4726e7c5b70750ee6f839b888637b66f12b6152d8b026726a77501c701f4fbb6"}
Aug 01 00:37:23.894465 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:23.894329    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf" event={"ID":"801862f1-32f0-4e77-84bf-ad4895e9f6ac","Type":"ContainerStarted","Data":"4f9c15f79a26a70a7a89c623413533f690d0c2c92cf50f0c3863edb38876f43b"}
Aug 01 00:37:25.905465 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:25.905438    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf_801862f1-32f0-4e77-84bf-ad4895e9f6ac/storage-initializer/0.log"
Aug 01 00:37:25.905832 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:25.905481    2584 generic.go:356] "Generic (PLEG): container finished" podID="801862f1-32f0-4e77-84bf-ad4895e9f6ac" containerID="4726e7c5b70750ee6f839b888637b66f12b6152d8b026726a77501c701f4fbb6" exitCode=1
Aug 01 00:37:25.905832 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:25.905558    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf" event={"ID":"801862f1-32f0-4e77-84bf-ad4895e9f6ac","Type":"ContainerDied","Data":"4726e7c5b70750ee6f839b888637b66f12b6152d8b026726a77501c701f4fbb6"}
Aug 01 00:37:26.917616 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:26.917589    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf_801862f1-32f0-4e77-84bf-ad4895e9f6ac/storage-initializer/0.log"
Aug 01 00:37:26.918022 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:26.917673    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf" event={"ID":"801862f1-32f0-4e77-84bf-ad4895e9f6ac","Type":"ContainerStarted","Data":"2dd3d6ab01a635f18349fba98b0430af864927ec4308e1db1d06ed233dcebc68"}
Aug 01 00:37:32.945250 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:32.945178    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf_801862f1-32f0-4e77-84bf-ad4895e9f6ac/storage-initializer/1.log"
Aug 01 00:37:32.945625 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:32.945608    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf_801862f1-32f0-4e77-84bf-ad4895e9f6ac/storage-initializer/0.log"
Aug 01 00:37:32.945681 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:32.945644    2584 generic.go:356] "Generic (PLEG): container finished" podID="801862f1-32f0-4e77-84bf-ad4895e9f6ac" containerID="2dd3d6ab01a635f18349fba98b0430af864927ec4308e1db1d06ed233dcebc68" exitCode=1
Aug 01 00:37:32.945718 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:32.945689    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf" event={"ID":"801862f1-32f0-4e77-84bf-ad4895e9f6ac","Type":"ContainerDied","Data":"2dd3d6ab01a635f18349fba98b0430af864927ec4308e1db1d06ed233dcebc68"}
Aug 01 00:37:32.945753 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:32.945744    2584 scope.go:117] "RemoveContainer" containerID="4726e7c5b70750ee6f839b888637b66f12b6152d8b026726a77501c701f4fbb6"
Aug 01 00:37:32.946216 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:32.946196    2584 scope.go:117] "RemoveContainer" containerID="4726e7c5b70750ee6f839b888637b66f12b6152d8b026726a77501c701f4fbb6"
Aug 01 00:37:32.959460 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:37:32.959422    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf_kserve-ci-e2e-test(801862f1-32f0-4e77-84bf-ad4895e9f6ac)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf" podUID="801862f1-32f0-4e77-84bf-ad4895e9f6ac"
Aug 01 00:37:32.959655 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:37:32.959513    2584 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf_kserve-ci-e2e-test_801862f1-32f0-4e77-84bf-ad4895e9f6ac_0 in pod sandbox 4f9c15f79a26a70a7a89c623413533f690d0c2c92cf50f0c3863edb38876f43b from index: no such id: '4726e7c5b70750ee6f839b888637b66f12b6152d8b026726a77501c701f4fbb6'" containerID="4726e7c5b70750ee6f839b888637b66f12b6152d8b026726a77501c701f4fbb6"
Aug 01 00:37:32.959792 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:32.959663    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4726e7c5b70750ee6f839b888637b66f12b6152d8b026726a77501c701f4fbb6"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf_kserve-ci-e2e-test_801862f1-32f0-4e77-84bf-ad4895e9f6ac_0 in pod sandbox 4f9c15f79a26a70a7a89c623413533f690d0c2c92cf50f0c3863edb38876f43b from index: no such id: '4726e7c5b70750ee6f839b888637b66f12b6152d8b026726a77501c701f4fbb6'"
Aug 01 00:37:32.966435 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:32.966402    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"]
Aug 01 00:37:33.022902 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:37:33.022858    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Aug 01 00:37:33.023133 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:37:33.022952    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/801862f1-32f0-4e77-84bf-ad4895e9f6ac-proxy-tls podName:801862f1-32f0-4e77-84bf-ad4895e9f6ac nodeName:}" failed. No retries permitted until 2026-08-01 00:37:33.522928598 +0000 UTC m=+3088.396572186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/801862f1-32f0-4e77-84bf-ad4895e9f6ac-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf" (UID: "801862f1-32f0-4e77-84bf-ad4895e9f6ac") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Aug 01 00:37:33.527666 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:37:33.527613    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Aug 01 00:37:33.527894 ip-10-0-137-252 kubenswrapper[2584]: E0801 00:37:33.527711    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/801862f1-32f0-4e77-84bf-ad4895e9f6ac-proxy-tls podName:801862f1-32f0-4e77-84bf-ad4895e9f6ac nodeName:}" failed. No retries permitted until 2026-08-01 00:37:34.527691399 +0000 UTC m=+3089.401334988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/801862f1-32f0-4e77-84bf-ad4895e9f6ac-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf" (UID: "801862f1-32f0-4e77-84bf-ad4895e9f6ac") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Aug 01 00:37:33.951532 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:33.951505    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf_801862f1-32f0-4e77-84bf-ad4895e9f6ac/storage-initializer/1.log"
Aug 01 00:37:34.080900 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.080876    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf_801862f1-32f0-4e77-84bf-ad4895e9f6ac/storage-initializer/1.log"
Aug 01 00:37:34.084421 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.084403    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:34.132207 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.132172    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tphvs\" (UniqueName: \"kubernetes.io/projected/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kube-api-access-tphvs\") pod \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") "
Aug 01 00:37:34.132207 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.132208    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kube-rbac-proxy-sar-config\") pod \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") "
Aug 01 00:37:34.132443 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.132258    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/801862f1-32f0-4e77-84bf-ad4895e9f6ac-proxy-tls\") pod \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") "
Aug 01 00:37:34.132443 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.132298    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kserve-provision-location\") pod \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\" (UID: \"801862f1-32f0-4e77-84bf-ad4895e9f6ac\") "
Aug 01 00:37:34.132633 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.132609    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "801862f1-32f0-4e77-84bf-ad4895e9f6ac" (UID: "801862f1-32f0-4e77-84bf-ad4895e9f6ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:37:34.132633 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.132615    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "801862f1-32f0-4e77-84bf-ad4895e9f6ac" (UID: "801862f1-32f0-4e77-84bf-ad4895e9f6ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:37:34.134331 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.134309    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/801862f1-32f0-4e77-84bf-ad4895e9f6ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "801862f1-32f0-4e77-84bf-ad4895e9f6ac" (UID: "801862f1-32f0-4e77-84bf-ad4895e9f6ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:37:34.134411 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.134373    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kube-api-access-tphvs" (OuterVolumeSpecName: "kube-api-access-tphvs") pod "801862f1-32f0-4e77-84bf-ad4895e9f6ac" (UID: "801862f1-32f0-4e77-84bf-ad4895e9f6ac"). InnerVolumeSpecName "kube-api-access-tphvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:37:34.233029 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.232919    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tphvs\" (UniqueName: \"kubernetes.io/projected/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kube-api-access-tphvs\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:37:34.233029 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.232994    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:37:34.233029 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.233007    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/801862f1-32f0-4e77-84bf-ad4895e9f6ac-proxy-tls\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:37:34.233029 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.233016    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/801862f1-32f0-4e77-84bf-ad4895e9f6ac-kserve-provision-location\") on node \"ip-10-0-137-252.ec2.internal\" DevicePath \"\""
Aug 01 00:37:34.957164 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.957134    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf_801862f1-32f0-4e77-84bf-ad4895e9f6ac/storage-initializer/1.log"
Aug 01 00:37:34.957576 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.957230    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"
Aug 01 00:37:34.957576 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.957268    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf" event={"ID":"801862f1-32f0-4e77-84bf-ad4895e9f6ac","Type":"ContainerDied","Data":"4f9c15f79a26a70a7a89c623413533f690d0c2c92cf50f0c3863edb38876f43b"}
Aug 01 00:37:34.957576 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.957331    2584 scope.go:117] "RemoveContainer" containerID="2dd3d6ab01a635f18349fba98b0430af864927ec4308e1db1d06ed233dcebc68"
Aug 01 00:37:34.991257 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.991213    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"]
Aug 01 00:37:34.992644 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:34.992620    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-655d56478-kbpcf"]
Aug 01 00:37:35.571441 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:37:35.571404    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="801862f1-32f0-4e77-84bf-ad4895e9f6ac" path="/var/lib/kubelet/pods/801862f1-32f0-4e77-84bf-ad4895e9f6ac/volumes"
Aug 01 00:39:15.293689 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:15.293656    2584 ???:1] "http: TLS handshake error from 10.0.136.174:46614: EOF"
Aug 01 00:39:15.300426 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:15.300397    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sjj4w_b717d75a-de98-4e7b-b86e-1af68be8f6a7/global-pull-secret-syncer/0.log"
Aug 01 00:39:15.423686 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:15.423653    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dhhpx_776b1ef2-f435-44ca-9f0e-90316affeef8/konnectivity-agent/0.log"
Aug 01 00:39:15.492696 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:15.492671    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-252.ec2.internal_e413b343bfdc4c93b43b646e37497322/haproxy/0.log"
Aug 01 00:39:19.787254 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:19.787222    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9pkhd_0283c9e2-278c-41a6-b9ca-3a40d2f5be7c/node-exporter/0.log"
Aug 01 00:39:19.809467 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:19.809444    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9pkhd_0283c9e2-278c-41a6-b9ca-3a40d2f5be7c/kube-rbac-proxy/0.log"
Aug 01 00:39:19.830201 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:19.830181    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9pkhd_0283c9e2-278c-41a6-b9ca-3a40d2f5be7c/init-textfile/0.log"
Aug 01 00:39:20.396567 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:20.396539    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75f9cb8c65-g2cxd_1fc0a74a-6417-4658-9992-85c374fc5649/telemeter-client/0.log"
Aug 01 00:39:20.418617 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:20.418587    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75f9cb8c65-g2cxd_1fc0a74a-6417-4658-9992-85c374fc5649/reload/0.log"
Aug 01 00:39:20.439440 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:20.439412    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75f9cb8c65-g2cxd_1fc0a74a-6417-4658-9992-85c374fc5649/kube-rbac-proxy/0.log"
Aug 01 00:39:21.884943 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:21.884914    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-wpznf_65c4d1ec-7577-4e31-bc3e-cfed4f74a062/networking-console-plugin/0.log"
Aug 01 00:39:22.379391 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:22.379361    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/1.log"
Aug 01 00:39:22.384583 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:22.384562    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b6g6c_c31e0617-9006-4866-91ab-4a72295b5771/console-operator/2.log"
Aug 01 00:39:23.281277 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.281242    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-t6zcg_c8df51d7-7157-4743-a324-f3ad06e16c81/volume-data-source-validator/0.log"
Aug 01 00:39:23.717484 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.717450    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"]
Aug 01 00:39:23.717814 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.717796    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801862f1-32f0-4e77-84bf-ad4895e9f6ac" containerName="storage-initializer"
Aug 01 00:39:23.717906 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.717818    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="801862f1-32f0-4e77-84bf-ad4895e9f6ac" containerName="storage-initializer"
Aug 01 00:39:23.717906 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.717831    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801862f1-32f0-4e77-84bf-ad4895e9f6ac" containerName="storage-initializer"
Aug 01 00:39:23.717906 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.717839    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="801862f1-32f0-4e77-84bf-ad4895e9f6ac" containerName="storage-initializer"
Aug 01 00:39:23.718085 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.717959    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="801862f1-32f0-4e77-84bf-ad4895e9f6ac" containerName="storage-initializer"
Aug 01 00:39:23.720998 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.720957    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:23.723686 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.723657    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7jcv6\"/\"default-dockercfg-mwjjj\""
Aug 01 00:39:23.723832 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.723703    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7jcv6\"/\"kube-root-ca.crt\""
Aug 01 00:39:23.723832 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.723752    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7jcv6\"/\"openshift-service-ca.crt\""
Aug 01 00:39:23.729090 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.728498    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"]
Aug 01 00:39:23.827112 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.827066    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/993de8a6-8d93-4b2d-84a4-5002d2c96405-proc\") pod \"perf-node-gather-daemonset-7pddd\" (UID: \"993de8a6-8d93-4b2d-84a4-5002d2c96405\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:23.827482 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.827144    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/993de8a6-8d93-4b2d-84a4-5002d2c96405-podres\") pod \"perf-node-gather-daemonset-7pddd\" (UID: \"993de8a6-8d93-4b2d-84a4-5002d2c96405\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:23.827482 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.827167    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/993de8a6-8d93-4b2d-84a4-5002d2c96405-sys\") pod \"perf-node-gather-daemonset-7pddd\" (UID: \"993de8a6-8d93-4b2d-84a4-5002d2c96405\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:23.827482 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.827187    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/993de8a6-8d93-4b2d-84a4-5002d2c96405-lib-modules\") pod \"perf-node-gather-daemonset-7pddd\" (UID: \"993de8a6-8d93-4b2d-84a4-5002d2c96405\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:23.827482 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.827213    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chr6w\" (UniqueName: \"kubernetes.io/projected/993de8a6-8d93-4b2d-84a4-5002d2c96405-kube-api-access-chr6w\") pod \"perf-node-gather-daemonset-7pddd\" (UID: \"993de8a6-8d93-4b2d-84a4-5002d2c96405\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:23.928492 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.928455    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/993de8a6-8d93-4b2d-84a4-5002d2c96405-podres\") pod \"perf-node-gather-daemonset-7pddd\" (UID: \"993de8a6-8d93-4b2d-84a4-5002d2c96405\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:23.928650 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.928497    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/993de8a6-8d93-4b2d-84a4-5002d2c96405-sys\") pod \"perf-node-gather-daemonset-7pddd\" (UID: \"993de8a6-8d93-4b2d-84a4-5002d2c96405\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:23.928650 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.928533    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/993de8a6-8d93-4b2d-84a4-5002d2c96405-lib-modules\") pod \"perf-node-gather-daemonset-7pddd\" (UID: \"993de8a6-8d93-4b2d-84a4-5002d2c96405\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:23.928650 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.928568    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chr6w\" (UniqueName: \"kubernetes.io/projected/993de8a6-8d93-4b2d-84a4-5002d2c96405-kube-api-access-chr6w\") pod \"perf-node-gather-daemonset-7pddd\" (UID: \"993de8a6-8d93-4b2d-84a4-5002d2c96405\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:23.928650 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.928604    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/993de8a6-8d93-4b2d-84a4-5002d2c96405-proc\") pod \"perf-node-gather-daemonset-7pddd\" (UID: \"993de8a6-8d93-4b2d-84a4-5002d2c96405\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:23.928650 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.928602    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/993de8a6-8d93-4b2d-84a4-5002d2c96405-sys\") pod \"perf-node-gather-daemonset-7pddd\" (UID: \"993de8a6-8d93-4b2d-84a4-5002d2c96405\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:23.928894 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.928648    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/993de8a6-8d93-4b2d-84a4-5002d2c96405-podres\") pod \"perf-node-gather-daemonset-7pddd\" (UID: \"993de8a6-8d93-4b2d-84a4-5002d2c96405\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:23.928894 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.928667    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/993de8a6-8d93-4b2d-84a4-5002d2c96405-proc\") pod \"perf-node-gather-daemonset-7pddd\" (UID: \"993de8a6-8d93-4b2d-84a4-5002d2c96405\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:23.928894 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.928707    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/993de8a6-8d93-4b2d-84a4-5002d2c96405-lib-modules\") pod \"perf-node-gather-daemonset-7pddd\" (UID: \"993de8a6-8d93-4b2d-84a4-5002d2c96405\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:23.936852 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:23.936828    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chr6w\" (UniqueName: \"kubernetes.io/projected/993de8a6-8d93-4b2d-84a4-5002d2c96405-kube-api-access-chr6w\") pod \"perf-node-gather-daemonset-7pddd\" (UID: \"993de8a6-8d93-4b2d-84a4-5002d2c96405\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:24.032725 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:24.032648    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:24.160223 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:24.160052    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"]
Aug 01 00:39:24.162780 ip-10-0-137-252 kubenswrapper[2584]: W0801 00:39:24.162742    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod993de8a6_8d93_4b2d_84a4_5002d2c96405.slice/crio-d2b0d378e8e0b15f56afe509c61190e218b30cd3f2543ee1b917d554bf6656bb WatchSource:0}: Error finding container d2b0d378e8e0b15f56afe509c61190e218b30cd3f2543ee1b917d554bf6656bb: Status 404 returned error can't find the container with id d2b0d378e8e0b15f56afe509c61190e218b30cd3f2543ee1b917d554bf6656bb
Aug 01 00:39:24.203581 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:24.203557    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zmrgg_f6ce1e39-8710-41b1-8fe8-c6ba227ad006/dns/0.log"
Aug 01 00:39:24.221015 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:24.220993    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zmrgg_f6ce1e39-8710-41b1-8fe8-c6ba227ad006/kube-rbac-proxy/0.log"
Aug 01 00:39:24.243186 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:24.243166    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hsjmv_1df3a7f4-80f0-4a56-a857-d59531130023/dns-node-resolver/0.log"
Aug 01 00:39:24.407322 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:24.407287    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd" event={"ID":"993de8a6-8d93-4b2d-84a4-5002d2c96405","Type":"ContainerStarted","Data":"112f89c14b67c06b993cd58ce2a946f536f88f1f5b548c4cc403e983dd023e3a"}
Aug 01 00:39:24.407707 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:24.407390    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:24.407707 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:24.407415    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd" event={"ID":"993de8a6-8d93-4b2d-84a4-5002d2c96405","Type":"ContainerStarted","Data":"d2b0d378e8e0b15f56afe509c61190e218b30cd3f2543ee1b917d554bf6656bb"}
Aug 01 00:39:24.423497 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:24.423455    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd" podStartSLOduration=1.423439749 podStartE2EDuration="1.423439749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:39:24.422398058 +0000 UTC m=+3199.296041669" watchObservedRunningTime="2026-08-01 00:39:24.423439749 +0000 UTC m=+3199.297083404"
Aug 01 00:39:24.677609 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:24.677532    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29759040-dmdz6_98479c90-4fef-4b82-af4f-d9bb7ab6fa6c/image-pruner/0.log"
Aug 01 00:39:24.773907 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:24.773865    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s9w8v_ce48bcb1-ccef-4183-af51-8a88c043cb64/node-ca/0.log"
Aug 01 00:39:25.789987 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:25.789934    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ghqfz_af99387b-4f11-481c-a86d-aa666c795d94/serve-healthcheck-canary/0.log"
Aug 01 00:39:26.119557 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:26.119471    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-rp75j_45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b/insights-operator/0.log"
Aug 01 00:39:26.119724 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:26.119670    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-rp75j_45edc69d-fbf6-46bf-8c4b-9aecc7c2f43b/insights-operator/1.log"
Aug 01 00:39:26.243914 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:26.243879    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x799c_59ff7f6d-a786-4afb-9fd6-90b59a73ffd9/kube-rbac-proxy/0.log"
Aug 01 00:39:26.260351 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:26.260322    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x799c_59ff7f6d-a786-4afb-9fd6-90b59a73ffd9/exporter/0.log"
Aug 01 00:39:26.276791 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:26.276761    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x799c_59ff7f6d-a786-4afb-9fd6-90b59a73ffd9/extractor/0.log"
Aug 01 00:39:28.400385 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:28.400350    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-674d959fb8-md78t_35f2b516-698c-4978-95cc-c1500eafdbfe/manager/0.log"
Aug 01 00:39:28.718751 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:28.718673    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-8588677cfb-482qx_ef58b45c-cd1b-4b63-b4bf-afee17ec88ff/manager/0.log"
Aug 01 00:39:28.858095 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:28.858066    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-2ldmx_ba2f18cd-6d28-4e57-aa42-f8d2f0e46f32/seaweedfs/0.log"
Aug 01 00:39:28.878086 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:28.878057    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-pjs54_1730bd87-0467-4e98-8651-8fdf1b6a995b/seaweedfs-tls-custom/0.log"
Aug 01 00:39:28.895728 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:28.895701    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-x6lk9_90812f05-db47-4035-93e8-d821f3723f3f/seaweedfs-tls-serving/0.log"
Aug 01 00:39:30.421270 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:30.421245    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-7pddd"
Aug 01 00:39:32.738400 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:32.738319    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-nmn97_51994692-c2a6-4b7e-a1c3-ca8255d407e8/kube-storage-version-migrator-operator/1.log"
Aug 01 00:39:32.739299 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:32.739280    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-nmn97_51994692-c2a6-4b7e-a1c3-ca8255d407e8/kube-storage-version-migrator-operator/0.log"
Aug 01 00:39:33.643341 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:33.643307    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7hjxn_3372e998-c5cb-4cd3-a17e-d8fcfec8a2fc/kube-multus/0.log"
Aug 01 00:39:33.857221 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:33.857196    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-glccn_054662f6-b4fc-47bf-afbc-dee3eec7a1a7/kube-multus-additional-cni-plugins/0.log"
Aug 01 00:39:33.873826 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:33.873803    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-glccn_054662f6-b4fc-47bf-afbc-dee3eec7a1a7/egress-router-binary-copy/0.log"
Aug 01 00:39:33.893540 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:33.893517    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-glccn_054662f6-b4fc-47bf-afbc-dee3eec7a1a7/cni-plugins/0.log"
Aug 01 00:39:33.912362 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:33.912335    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-glccn_054662f6-b4fc-47bf-afbc-dee3eec7a1a7/bond-cni-plugin/0.log"
Aug 01 00:39:33.929929 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:33.929908    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-glccn_054662f6-b4fc-47bf-afbc-dee3eec7a1a7/routeoverride-cni/0.log"
Aug 01 00:39:33.945622 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:33.945601    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-glccn_054662f6-b4fc-47bf-afbc-dee3eec7a1a7/whereabouts-cni-bincopy/0.log"
Aug 01 00:39:33.962369 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:33.962347    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-glccn_054662f6-b4fc-47bf-afbc-dee3eec7a1a7/whereabouts-cni/0.log"
Aug 01 00:39:34.227020 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:34.226919    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hsnnq_9d3952c7-cede-4f37-aba3-9b96b84eaeb6/network-metrics-daemon/0.log"
Aug 01 00:39:34.242228 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:34.242203    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hsnnq_9d3952c7-cede-4f37-aba3-9b96b84eaeb6/kube-rbac-proxy/0.log"
Aug 01 00:39:35.213253 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:35.213226    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khzpq_52e351d7-6ba9-4265-bf15-4fcc3604a693/ovn-controller/0.log"
Aug 01 00:39:35.247778 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:35.247737    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khzpq_52e351d7-6ba9-4265-bf15-4fcc3604a693/ovn-acl-logging/0.log"
Aug 01 00:39:35.296279 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:35.296247    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khzpq_52e351d7-6ba9-4265-bf15-4fcc3604a693/kube-rbac-proxy-node/0.log"
Aug 01 00:39:35.327944 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:35.327912    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khzpq_52e351d7-6ba9-4265-bf15-4fcc3604a693/kube-rbac-proxy-ovn-metrics/0.log"
Aug 01 00:39:35.342877 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:35.342849    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khzpq_52e351d7-6ba9-4265-bf15-4fcc3604a693/northd/0.log"
Aug 01 00:39:35.359102 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:35.359074    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khzpq_52e351d7-6ba9-4265-bf15-4fcc3604a693/nbdb/0.log"
Aug 01 00:39:35.374871 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:35.374841    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khzpq_52e351d7-6ba9-4265-bf15-4fcc3604a693/sbdb/0.log"
Aug 01 00:39:35.412406 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:35.412376    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khzpq_52e351d7-6ba9-4265-bf15-4fcc3604a693/ovnkube-controller/0.log"
Aug 01 00:39:36.630521 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:36.630479    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qpmwv_6ca51531-7776-413a-98b3-aebadd923730/network-check-target-container/0.log"
Aug 01 00:39:37.468468 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:37.468438    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-c6n68_d23c6534-149f-4a0b-baea-af7e105c571c/iptables-alerter/0.log"
Aug 01 00:39:38.086251 ip-10-0-137-252 kubenswrapper[2584]: I0801 00:39:38.086224    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-q7xzv_3079ae51-76cf-4bde-90bf-05e1396256b3/tuned/0.log"