Aug 04 17:58:12.249277 ip-10-0-139-237 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Aug 04 17:58:12.249289 ip-10-0-139-237 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Aug 04 17:58:12.249297 ip-10-0-139-237 systemd[1]: kubelet.service: Failed with result 'resources'.
Aug 04 17:58:12.249577 ip-10-0-139-237 systemd[1]: Failed to start Kubernetes Kubelet.
Aug 04 17:58:22.368568 ip-10-0-139-237 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Aug 04 17:58:22.368584 ip-10-0-139-237 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a0d01982ea20451cb328523aed83d2c0 --
Aug 04 17:59:56.913577 ip-10-0-139-237 systemd[1]: Starting Kubernetes Kubelet...
Aug 04 17:59:57.338989 ip-10-0-139-237 kubenswrapper[2543]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 17:59:57.338989 ip-10-0-139-237 kubenswrapper[2543]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 04 17:59:57.338989 ip-10-0-139-237 kubenswrapper[2543]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 17:59:57.338989 ip-10-0-139-237 kubenswrapper[2543]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 04 17:59:57.338989 ip-10-0-139-237 kubenswrapper[2543]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 17:59:57.341647 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.341585    2543 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 04 17:59:57.345425 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345411    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:57.345459 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345426    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:57.345459 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345431    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:57.345459 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345434    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:57.345459 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345438    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:57.345459 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345441    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:57.345459 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345444    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:57.345459 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345446    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:57.345459 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345449    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:57.345459 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345452    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:57.345459 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345455    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:57.345459 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345458    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:57.345459 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345461    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:57.345459 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345463    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345479    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345482    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345485    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345487    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345490    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345492    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345495    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345498    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345507    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345510    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345513    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345516    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345518    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345521    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345523    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345525    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345528    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345530    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345533    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:57.345773 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345535    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345537    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345540    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345542    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345545    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345547    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345549    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345552    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345555    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345557    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345560    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345562    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345565    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345568    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345576    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345579    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345582    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345584    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345587    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:57.346233 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345591    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345595    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345598    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345601    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345603    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345606    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345609    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345612    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345615    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345619    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345622    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345624    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345627    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345629    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345632    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345634    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345637    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345639    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345642    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:57.346698 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345645    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345649    2543 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345651    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345654    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345657    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345660    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345662    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345665    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345667    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345675    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345678    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345681    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345684    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345686    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345688    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345690    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345693    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345695    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345698    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345700    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:57.347152 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345702    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345705    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345707    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345710    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345712    2543 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345715    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345717    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.345720    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346416    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346423    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346426    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346429    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346432    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346435    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346438    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346440    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346443    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346445    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346448    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346450    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:57.347657 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346453    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346455    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346459    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346461    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346464    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346467    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346470    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346474    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346476    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346479    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346481    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346484    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346486    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346489    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346492    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346495    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346497    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346502    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346505    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:57.348132 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346508    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346511    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346514    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346516    2543 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346519    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346521    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346524    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346528    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346531    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346534    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346536    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346539    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346541    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346544    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346547    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346550    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346553    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346555    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346557    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346560    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:57.348609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346562    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346565    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346568    2543 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346570    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346572    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346575    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346577    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346579    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346582    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346584    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346587    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346590    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346592    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346595    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346597    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346600    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346602    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346605    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346608    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346611    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:57.349079 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346613    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346616    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346619    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346621    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346624    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346626    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346629    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346631    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346634    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346637    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346639    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346642    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346644    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346647    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346651    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346654    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346656    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346659    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346662    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346664    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:57.349563 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346668    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346672    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346674    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346677    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346680    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346683    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346686    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.346688    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347074    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347079    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347082    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347085    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347088    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347091    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347093    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347096    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347098    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347101    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347103    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:57.350039 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347106    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347108    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347111    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347114    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347116    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347119    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347121    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347124    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347126    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347129    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347131    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347134    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347136    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347139    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347141    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347144    2543 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347146    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347149    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347151    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347154    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:57.350516 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347156    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347160    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347163    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347166    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347168    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347171    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347173    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347176    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347178    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347181    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347184    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347186    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347188    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347192    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347195    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347199    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347202    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347204    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347207    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:57.350995 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347210    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347212    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347215    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347219    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347222    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347225    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347228    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347231    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347234    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347237    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347240    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347242    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347245    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347248    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347250    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347252    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347255    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347258    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347260    2543 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347263    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:57.351505 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347265    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347267    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347270    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347272    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347276    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347278    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347281    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347283    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347285    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347288    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347291    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347293    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347296    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347298    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347301    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347303    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347320    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347325    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347329    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347333    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:57.352007 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347336    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347339    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347342    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347345    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347347    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347351    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347355    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347358    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347361    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347420    2543 flags.go:64] FLAG: --address="0.0.0.0"
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347427    2543 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347432    2543 flags.go:64] FLAG: --anonymous-auth="true"
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347436    2543 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347440    2543 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347443    2543 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347447    2543 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347451    2543 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347454    2543 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347456    2543 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347460    2543 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347463    2543 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347466    2543 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 04 17:59:57.352486 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347469    2543 flags.go:64] FLAG: --cgroup-root=""
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347472    2543 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347475    2543 flags.go:64] FLAG: --client-ca-file=""
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347478    2543 flags.go:64] FLAG: --cloud-provider="external"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347481    2543 flags.go:64] FLAG: --cluster-dns="[]"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347486    2543 flags.go:64] FLAG: --cluster-domain=""
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347489    2543 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347492    2543 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347495    2543 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347498    2543 flags.go:64] FLAG: --container-log-max-files="5"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347502    2543 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347505    2543 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347508    2543 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347511    2543 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347514    2543 flags.go:64] FLAG: --contention-profiling="false"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347516    2543 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347519    2543 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347522    2543 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347525    2543 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347534    2543 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347537    2543 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347540    2543 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347542    2543 flags.go:64] FLAG: --enable-load-reader="false"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347545    2543 flags.go:64] FLAG: --enable-server="true"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347548    2543 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 04 17:59:57.352996 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347552    2543 flags.go:64] FLAG: --event-burst="100"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347555    2543 flags.go:64] FLAG: --event-qps="50"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347558    2543 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347560    2543 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347563    2543 flags.go:64] FLAG: --eviction-hard=""
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347566    2543 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347569    2543 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347572    2543 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347575    2543 flags.go:64] FLAG: --eviction-soft=""
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347578    2543 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347581    2543 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347583    2543 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347586    2543 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347589    2543 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347592    2543 flags.go:64] FLAG: --fail-swap-on="true"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347596    2543 flags.go:64] FLAG: --feature-gates=""
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347600    2543 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347603    2543 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347605    2543 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347608    2543 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347611    2543 flags.go:64] FLAG: --healthz-port="10248"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347614    2543 flags.go:64] FLAG: --help="false"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347617    2543 flags.go:64] FLAG: --hostname-override="ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347620    2543 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 04 17:59:57.353688 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347623    2543 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347626    2543 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347629    2543 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347632    2543 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347635    2543 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347638    2543 flags.go:64] FLAG: --image-service-endpoint=""
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347641    2543 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347643    2543 flags.go:64] FLAG: --kube-api-burst="100"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347646    2543 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347649    2543 flags.go:64] FLAG: --kube-api-qps="50"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347652    2543 flags.go:64] FLAG: --kube-reserved=""
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347654    2543 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347657    2543 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347660    2543 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347663    2543 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347666    2543 flags.go:64] FLAG: --lock-file=""
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347669    2543 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347672    2543 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347675    2543 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347680    2543 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347683    2543 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347686    2543 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347688    2543 flags.go:64] FLAG: --logging-format="text"
Aug 04 17:59:57.354272 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347692    2543 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347695    2543 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347698    2543 flags.go:64] FLAG: --manifest-url=""
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347701    2543 flags.go:64] FLAG: --manifest-url-header=""
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347705    2543 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347707    2543 flags.go:64] FLAG: --max-open-files="1000000"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347711    2543 flags.go:64] FLAG: --max-pods="110"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347715    2543 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347718    2543 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347720    2543 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347723    2543 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347726    2543 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347729    2543 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347731    2543 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347738    2543 flags.go:64] FLAG: --node-status-max-images="50"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347740    2543 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347743    2543 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347746    2543 flags.go:64] FLAG: --pod-cidr=""
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347749    2543 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347754    2543 flags.go:64] FLAG: --pod-manifest-path=""
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347756    2543 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347759    2543 flags.go:64] FLAG: --pods-per-core="0"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347762    2543 flags.go:64] FLAG: --port="10250"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347765    2543 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 04 17:59:57.354957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347768    2543 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-001e3ca6aebf26c57"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347771    2543 flags.go:64] FLAG: --qos-reserved=""
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347773    2543 flags.go:64] FLAG: --read-only-port="10255"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347777    2543 flags.go:64] FLAG: --register-node="true"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347779    2543 flags.go:64] FLAG: --register-with-taints=""
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347783    2543 flags.go:64] FLAG: --registry-burst="10"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347787    2543 flags.go:64] FLAG: --registry-qps="5"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347790    2543 flags.go:64] FLAG: --reserved-cpus=""
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347794    2543 flags.go:64] FLAG: --reserved-memory=""
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347798    2543 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347801    2543 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347804    2543 flags.go:64] FLAG: --rotate-certificates="false"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347806    2543 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347809    2543 flags.go:64] FLAG: --runonce="false"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347812    2543 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347815    2543 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347818    2543 flags.go:64] FLAG: --seccomp-default="false"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347821    2543 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347823    2543 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347826    2543 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347829    2543 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347832    2543 flags.go:64] FLAG: --storage-driver-password="root"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347834    2543 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347837    2543 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347840    2543 flags.go:64] FLAG: --storage-driver-user="root"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347843    2543 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 04 17:59:57.356029 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347846    2543 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347849    2543 flags.go:64] FLAG: --system-cgroups=""
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347852    2543 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347856    2543 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347859    2543 flags.go:64] FLAG: --tls-cert-file=""
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347862    2543 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347866    2543 flags.go:64] FLAG: --tls-min-version=""
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347868    2543 flags.go:64] FLAG: --tls-private-key-file=""
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347871    2543 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347874    2543 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347877    2543 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347880    2543 flags.go:64] FLAG: --v="2"
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347884    2543 flags.go:64] FLAG: --version="false"
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347888    2543 flags.go:64] FLAG: --vmodule=""
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347892    2543 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.347896    2543 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.347998    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348002    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348004    2543 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348007    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348010    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348012    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348014    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:57.357112 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348017    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348019    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348022    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348024    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348027    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348029    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348032    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348034    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348037    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348039    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348042    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348044    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348047    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348049    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348052    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348054    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348057    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348059    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348062    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348064    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:57.358093 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348067    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348071    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348074    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348077    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348080    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348083    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348085    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348088    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348090    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348093    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348096    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348098    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348101    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348103    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348106    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348108    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348111    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348114    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348116    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348120    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:57.359005 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348123    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348125    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348127    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348130    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348133    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348136    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348138    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348141    2543 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348145    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348149    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348152    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348154    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348157    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348161    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348164    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348167    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348170    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348173    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348175    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:57.359864 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348178    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348180    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348183    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348185    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348188    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348190    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348193    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348195    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348198    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348200    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348204    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348207    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348210    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348213    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348215    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348218    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348220    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348223    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348225    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:57.360484 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348228    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:57.361126 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348230    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:57.361126 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348233    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:57.361126 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348235    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:57.361126 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348237    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:57.361126 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348241    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:57.361126 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348245    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:57.361126 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348250    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:57.361126 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348253    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:57.361126 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348258    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:57.361126 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348261    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:57.361126 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348264    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:57.361126 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348266    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:57.361126 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.348269    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:57.361126 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.348939    2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 17:59:57.361126 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.357774    2543 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.357796    2543 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.357821    2543 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.357834    2543 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358045    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358053    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358058    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358064    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358069    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358075    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358085    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358089    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358094    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358099    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358103    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358108    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358113    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358118    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358123    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358127    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:57.361552 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358133    2543 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358138    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358146    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358151    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358156    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358160    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358165    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358170    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358175    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358179    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358185    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358189    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358194    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358210    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358222    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358231    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358237    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358244    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358248    2543 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:57.362050 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358253    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358260    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358265    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358270    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358275    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358280    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358284    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358289    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358298    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358303    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358331    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358336    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358340    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358345    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358350    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358354    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358359    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358363    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358367    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:57.362515 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358372    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358386    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358391    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358395    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358400    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358405    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358409    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358414    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358426    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358431    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358436    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358440    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358445    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358449    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358457    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358462    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358466    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358470    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358474    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358479    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:57.362980 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358483    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358489    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358493    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358498    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358502    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358507    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358515    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358520    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358524    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358529    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358533    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358543    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358548    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358552    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358557    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358561    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358566    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358570    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358578    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:57.363486 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358583    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:57.364011 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358588    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:57.364011 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358602    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:57.364011 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358609    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:57.364011 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358614    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:57.364011 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.358618    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:57.364011 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.358627    2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 17:59:57.364011 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359094    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:57.364011 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359104    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:57.364011 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359114    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:57.364011 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359118    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:57.364011 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359125    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:57.364011 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359131    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:57.364011 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359137    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:57.364011 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359141    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359146    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359150    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359155    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359159    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359164    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359168    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359172    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359186    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359190    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359195    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359199    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359204    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359208    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359213    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359217    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359222    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359226    2543 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359230    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359235    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:57.364609 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359244    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359256    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359261    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359265    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359269    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359274    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359278    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359283    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359287    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359291    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359296    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359300    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359304    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359327    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359332    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359336    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359340    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359345    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359349    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:57.365157 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359354    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359358    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359363    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359367    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359372    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359377    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359387    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359392    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359397    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359401    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359406    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359410    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359415    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359420    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359425    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359437    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359442    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359446    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359455    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359459    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:57.365759 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359464    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359471    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359476    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359481    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359486    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359491    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359496    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359507    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359512    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359516    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359520    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359529    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359533    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359537    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359542    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359547    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359551    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359555    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359560    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:57.366245 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359564    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:57.366763 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359568    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:57.366763 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359572    2543 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:57.366763 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359577    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:57.366763 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359585    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:57.366763 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359590    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:57.366763 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359594    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:57.366763 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359598    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:57.366763 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359603    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:57.366763 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359615    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:57.366763 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359619    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:57.366763 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359624    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:57.366763 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359628    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:57.366763 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:57.359633    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:57.366763 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.359641    2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 17:59:57.366763 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.359930    2543 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 04 17:59:57.367125 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.365473    2543 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 04 17:59:57.367342 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.367293    2543 server.go:1019] "Starting client certificate rotation"
Aug 04 17:59:57.367424 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.367407    2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 17:59:57.367455 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.367442    2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 17:59:57.370073 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.370057    2543 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 17:59:57.373158 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.373142    2543 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 17:59:57.390082 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.390067    2543 log.go:25] "Validated CRI v1 runtime API"
Aug 04 17:59:57.392151 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.392138    2543 log.go:25] "Validated CRI v1 image API"
Aug 04 17:59:57.392985 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.392972    2543 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 04 17:59:57.395281 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.395256    2543 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c2cacc04-3f28-47f2-838d-72546691df4d:/dev/nvme0n1p4 fde10158-f303-4018-8f29-c74887e27369:/dev/nvme0n1p3]
Aug 04 17:59:57.395377 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.395279    2543 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 04 17:59:57.400327 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.400296    2543 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 17:59:57.401939 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.401459    2543 manager.go:216] Machine: {Timestamp:2026-08-04 17:59:57.40003739 +0000 UTC m=+0.370681722 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201740 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25c98925f072eb39218e3a3498072e SystemUUID:ec25c989-25f0-72eb-3921-8e3a3498072e BootID:a0d01982-ea20-451c-b328-523aed83d2c0 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c9:5f:d3:e9:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c9:5f:d3:e9:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:1c:ac:04:ba:ce Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 04 17:59:57.401939 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.401932    2543 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 04 17:59:57.402082 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.402019    2543 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 04 17:59:57.405417 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.405397    2543 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 04 17:59:57.405571 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.405419    2543 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-237.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 04 17:59:57.406338 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.406328    2543 topology_manager.go:138] "Creating topology manager with none policy"
Aug 04 17:59:57.406376 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.406339    2543 container_manager_linux.go:309] "Creating device plugin manager"
Aug 04 17:59:57.406376 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.406346    2543 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 17:59:57.406376 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.406358    2543 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 17:59:57.407553 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.407543    2543 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 04 17:59:57.411818 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.411807    2543 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 17:59:57.411924 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.411915    2543 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 04 17:59:57.411970 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.411941    2543 kubelet.go:486] "Attempting to sync node with API server"
Aug 04 17:59:57.411970 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.411950    2543 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 04 17:59:57.411970 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.411969    2543 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 04 17:59:57.412095 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.411981    2543 kubelet.go:398] "Adding apiserver pod source"
Aug 04 17:59:57.412095 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.412001    2543 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 04 17:59:57.412358 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.412344    2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 17:59:57.412412 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.412384    2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 17:59:57.416050 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.416027    2543 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 04 17:59:57.417937 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.417924    2543 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 04 17:59:57.418002 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.417953    2543 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 04 17:59:57.419245 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.419233    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 04 17:59:57.419286 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.419248    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 04 17:59:57.419286 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.419254    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 04 17:59:57.419286 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.419260    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 04 17:59:57.419286 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.419268    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 04 17:59:57.419286 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.419273    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 04 17:59:57.419286 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.419280    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 04 17:59:57.419286 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.419285    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 04 17:59:57.419487 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.419291    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 04 17:59:57.419487 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.419296    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 04 17:59:57.419487 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.419305    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 04 17:59:57.419487 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.419327    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 04 17:59:57.421215 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.421202    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 04 17:59:57.421271 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.421217    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 04 17:59:57.422830 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.422813    2543 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-237.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 04 17:59:57.424374 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.424352    2543 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-237.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 04 17:59:57.424420 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.424351    2543 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 04 17:59:57.424566 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.424555    2543 server.go:1268] "Started kubelet"
Aug 04 17:59:57.424657 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.424639    2543 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 04 17:59:57.424785 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.424742    2543 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 04 17:59:57.424841 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.424806    2543 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 04 17:59:57.426035 ip-10-0-139-237 systemd[1]: Started Kubernetes Kubelet.
Aug 04 17:59:57.426330 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.426302    2543 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 04 17:59:57.428818 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.427899    2543 server.go:310] "Adding debug handlers to kubelet server"
Aug 04 17:59:57.429769 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.429751    2543 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 04 17:59:57.434255 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.434236    2543 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 04 17:59:57.434255 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.434244    2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 04 17:59:57.434417 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.434301    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Aug 04 17:59:57.435019 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.435002    2543 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 04 17:59:57.435098 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.435005    2543 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 04 17:59:57.435098 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.435048    2543 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 04 17:59:57.435182 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.435107    2543 reconstruct.go:97] "Volume reconstruction finished"
Aug 04 17:59:57.435182 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.435116    2543 reconciler.go:29] "Reconciler: start to sync state"
Aug 04 17:59:57.435268 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.435176    2543 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 04 17:59:57.435268 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.435194    2543 factory.go:55] Registering systemd factory
Aug 04 17:59:57.435268 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.435203    2543 factory.go:223] Registration of the systemd container factory successfully
Aug 04 17:59:57.435456 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.435374    2543 factory.go:153] Registering CRI-O factory
Aug 04 17:59:57.435456 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.435384    2543 factory.go:223] Registration of the crio container factory successfully
Aug 04 17:59:57.435456 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.435404    2543 factory.go:103] Registering Raw factory
Aug 04 17:59:57.435456 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.435417    2543 manager.go:1196] Started watching for new ooms in manager
Aug 04 17:59:57.435801 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.435790    2543 manager.go:349] Starting recovery of all containers
Aug 04 17:59:57.445168 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.445139    2543 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Aug 04 17:59:57.445591 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.445565    2543 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-237.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Aug 04 17:59:57.447278 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.447260    2543 manager.go:354] Recovery completed
Aug 04 17:59:57.448577 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.445167    2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-237.ec2.internal.18c8ac06caca1b70  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-237.ec2.internal,UID:ip-10-0-139-237.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-237.ec2.internal,},FirstTimestamp:2026-08-04 17:59:57.42442584 +0000 UTC m=+0.395070172,LastTimestamp:2026-08-04 17:59:57.42442584 +0000 UTC m=+0.395070172,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-237.ec2.internal,}"
Aug 04 17:59:57.452880 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.452868    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:57.455015 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.454989    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:57.455069 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.455023    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:57.455069 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.455033    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:57.455477 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.455463    2543 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 04 17:59:57.455477 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.455475    2543 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 04 17:59:57.455591 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.455488    2543 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 17:59:57.455591 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.455532    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:57.456879 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.456865    2543 policy_none.go:49] "None policy: Start"
Aug 04 17:59:57.456879 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.456880    2543 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 04 17:59:57.456989 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.456892    2543 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 04 17:59:57.456989 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.456849    2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-237.ec2.internal.18c8ac06cc9cd2f1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-237.ec2.internal,UID:ip-10-0-139-237.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-237.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-237.ec2.internal,},FirstTimestamp:2026-08-04 17:59:57.455012593 +0000 UTC m=+0.425656923,LastTimestamp:2026-08-04 17:59:57.455012593 +0000 UTC m=+0.425656923,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-237.ec2.internal,}"
Aug 04 17:59:57.457349 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.457333    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:57.457388 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.457365    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:57.457388 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.457379    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:57.458011 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.458000    2543 policy_none.go:47] "Start"
Aug 04 17:59:57.467479 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.467424    2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-237.ec2.internal.18c8ac06cc9d11ba  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-237.ec2.internal,UID:ip-10-0-139-237.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-237.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-237.ec2.internal,},FirstTimestamp:2026-08-04 17:59:57.455028666 +0000 UTC m=+0.425672997,LastTimestamp:2026-08-04 17:59:57.455028666 +0000 UTC m=+0.425672997,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-237.ec2.internal,}"
Aug 04 17:59:57.479583 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.479511    2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-237.ec2.internal.18c8ac06cc9d3339  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-237.ec2.internal,UID:ip-10-0-139-237.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-237.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-237.ec2.internal,},FirstTimestamp:2026-08-04 17:59:57.455037241 +0000 UTC m=+0.425681572,LastTimestamp:2026-08-04 17:59:57.455037241 +0000 UTC m=+0.425681572,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-237.ec2.internal,}"
Aug 04 17:59:57.492832 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.492778    2543 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-237.ec2.internal.18c8ac06cc9cd2f1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-237.ec2.internal.18c8ac06cc9cd2f1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-237.ec2.internal,UID:ip-10-0-139-237.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-237.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-237.ec2.internal,},FirstTimestamp:2026-08-04 17:59:57.455012593 +0000 UTC m=+0.425656923,LastTimestamp:2026-08-04 17:59:57.457349375 +0000 UTC m=+0.427993709,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-237.ec2.internal,}"
Aug 04 17:59:57.507010 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.495879    2543 manager.go:337] "Starting Device Plugin manager"
Aug 04 17:59:57.507010 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.495901    2543 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 04 17:59:57.507010 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.495909    2543 server.go:87] "Starting device plugin registration server"
Aug 04 17:59:57.507010 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.496083    2543 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 04 17:59:57.507010 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.496092    2543 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 04 17:59:57.507010 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.496197    2543 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 04 17:59:57.507010 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.496273    2543 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 04 17:59:57.507010 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.496280    2543 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 04 17:59:57.507010 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.496749    2543 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 04 17:59:57.507010 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.496791    2543 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-237.ec2.internal\" not found"
Aug 04 17:59:57.507010 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.503710    2543 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-237.ec2.internal.18c8ac06cc9d11ba\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-237.ec2.internal.18c8ac06cc9d11ba  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-237.ec2.internal,UID:ip-10-0-139-237.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-237.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-237.ec2.internal,},FirstTimestamp:2026-08-04 17:59:57.455028666 +0000 UTC m=+0.425672997,LastTimestamp:2026-08-04 17:59:57.457372573 +0000 UTC m=+0.428016904,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-237.ec2.internal,}"
Aug 04 17:59:57.507010 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.505753    2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4hl9g"
Aug 04 17:59:57.513892 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.513878    2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4hl9g"
Aug 04 17:59:57.515330 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.515261    2543 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-237.ec2.internal.18c8ac06cc9d3339\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-237.ec2.internal.18c8ac06cc9d3339  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-237.ec2.internal,UID:ip-10-0-139-237.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-237.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-237.ec2.internal,},FirstTimestamp:2026-08-04 17:59:57.455037241 +0000 UTC m=+0.425681572,LastTimestamp:2026-08-04 17:59:57.457385001 +0000 UTC m=+0.428029333,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-237.ec2.internal,}"
Aug 04 17:59:57.567695 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.567654    2543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 04 17:59:57.568725 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.568705    2543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 04 17:59:57.568800 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.568737    2543 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 04 17:59:57.568800 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.568755    2543 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 04 17:59:57.568800 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.568786    2543 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 04 17:59:57.571350 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.571323    2543 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:57.596197 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.596154    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:57.596857 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.596829    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:57.596857 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.596854    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:57.596975 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.596868    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:57.596975 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.596892    2543 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.606488 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.606474    2543 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.606540 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.606493    2543 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-237.ec2.internal\": node \"ip-10-0-139-237.ec2.internal\" not found"
Aug 04 17:59:57.630261 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.630239    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Aug 04 17:59:57.668967 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.668947    2543 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal"]
Aug 04 17:59:57.669045 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.668997    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:57.669675 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.669663    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:57.669754 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.669688    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:57.669754 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.669702    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:57.671042 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.671027    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:57.671156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.671141    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.671199 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.671170    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:57.671637 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.671615    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:57.671696 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.671641    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:57.671696 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.671652    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:57.671696 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.671618    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:57.671779 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.671714    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:57.671779 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.671726    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:57.672720 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.672705    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.672772 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.672737    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:57.673464 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.673447    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:57.673548 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.673477    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:57.673548 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.673489    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:57.702008 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.701985    2543 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-237.ec2.internal\" not found" node="ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.706135 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.706118    2543 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-237.ec2.internal\" not found" node="ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.730528 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.730509    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Aug 04 17:59:57.736839 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.736818    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a97709841e9032d4d37a1c8319191f4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal\" (UID: \"a97709841e9032d4d37a1c8319191f4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.736943 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.736842    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a97709841e9032d4d37a1c8319191f4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal\" (UID: \"a97709841e9032d4d37a1c8319191f4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.736943 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.736858    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/28c305bdf94bdf64315ff2bce273b65b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-237.ec2.internal\" (UID: \"28c305bdf94bdf64315ff2bce273b65b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.831177 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.831159    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Aug 04 17:59:57.837508 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.837487    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a97709841e9032d4d37a1c8319191f4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal\" (UID: \"a97709841e9032d4d37a1c8319191f4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.837569 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.837524    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a97709841e9032d4d37a1c8319191f4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal\" (UID: \"a97709841e9032d4d37a1c8319191f4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.837569 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.837548    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a97709841e9032d4d37a1c8319191f4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal\" (UID: \"a97709841e9032d4d37a1c8319191f4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.837569 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.837562    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/28c305bdf94bdf64315ff2bce273b65b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-237.ec2.internal\" (UID: \"28c305bdf94bdf64315ff2bce273b65b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.837656 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.837579    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/28c305bdf94bdf64315ff2bce273b65b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-237.ec2.internal\" (UID: \"28c305bdf94bdf64315ff2bce273b65b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.837656 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:57.837595    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a97709841e9032d4d37a1c8319191f4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal\" (UID: \"a97709841e9032d4d37a1c8319191f4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:57.931844 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:57.931805    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Aug 04 17:59:58.005340 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.005320    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:58.009254 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.009242    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:58.032912 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:58.032895    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Aug 04 17:59:58.133361 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:58.133345    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Aug 04 17:59:58.233827 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:58.233791    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Aug 04 17:59:58.302758 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.302743    2543 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:58.334610 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.334586    2543 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:58.344266 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.344249    2543 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:58.366771 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.366749    2543 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 04 17:59:58.366878 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.366855    2543 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 04 17:59:58.366927 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.366882    2543 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 04 17:59:58.366927 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.366909    2543 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 04 17:59:58.366927 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:58.366908    2543 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://aca7f8fa9a1bb4c3d8246f8354ad9f18-f90a26d5c6137d70.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.139.237:55556->54.144.74.27:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:58.367019 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.366937    2543 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal"
Aug 04 17:59:58.392860 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.392845    2543 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 17:59:58.413092 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.413080    2543 apiserver.go:52] "Watching apiserver"
Aug 04 17:59:58.424121 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.424102    2543 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 04 17:59:58.424470 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.424446    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-bcz4n","openshift-dns/node-resolver-p52sc","openshift-image-registry/node-ca-z8dv8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal","openshift-multus/multus-additional-cni-plugins-2zthc","openshift-ovn-kubernetes/ovnkube-node-tkvl9","kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb","openshift-cluster-node-tuning-operator/tuned-hbksj","openshift-multus/multus-p4xjn","openshift-multus/network-metrics-daemon-r27tq","openshift-network-diagnostics/network-check-target-jzdgf","openshift-network-operator/iptables-alerter-qnwcg","kube-system/global-pull-secret-syncer-gsbtv"]
Aug 04 17:59:58.429148 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.429125    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.431055 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.431038    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.432120 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.432101    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 04 17:59:58.432230 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.432148    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 04 17:59:58.432230 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.432154    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 04 17:59:58.432230 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.432213    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4sgws\""
Aug 04 17:59:58.432971 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.432955    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.433620 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.433603    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 04 17:59:58.433717 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.433605    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ddwcv\""
Aug 04 17:59:58.433717 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.433633    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 04 17:59:58.434388 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.434369    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.434710 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.434691    2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 04 17:59:58.435525 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.435504    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 04 17:59:58.435645 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.435629    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 04 17:59:58.435716 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.435700    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 04 17:59:58.435782 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.435737    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 04 17:59:58.435989 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.435938    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 04 17:59:58.436087 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.436061    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nl6qp\""
Aug 04 17:59:58.436087 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.436071    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bcz4n"
Aug 04 17:59:58.436943 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.436926    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 04 17:59:58.437957 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.437941    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z8dv8"
Aug 04 17:59:58.438085 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.438032    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 04 17:59:58.438085 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.438044    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 04 17:59:58.438317 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.438084    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 04 17:59:58.438379 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.438317    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 04 17:59:58.439376 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.439352    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p52sc"
Aug 04 17:59:58.439668 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.439647    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 04 17:59:58.439763 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.439672    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kg7f4\""
Aug 04 17:59:58.439986 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.439968    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 04 17:59:58.440076 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440013    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 04 17:59:58.440136 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440090    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5kvpk\""
Aug 04 17:59:58.440185 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440167    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-host\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.440227 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440202    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d3889c85-d79f-4629-94bb-c3fe1c1471ff-os-release\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.440271 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440232    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2tqx\" (UniqueName: \"kubernetes.io/projected/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-kube-api-access-q2tqx\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.440271 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440257    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qwjq\" (UniqueName: \"kubernetes.io/projected/d520936e-622a-4cf8-852f-aaa274029d02-kube-api-access-4qwjq\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.440391 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440295    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3889c85-d79f-4629-94bb-c3fe1c1471ff-cnibin\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.440391 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440320    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3889c85-d79f-4629-94bb-c3fe1c1471ff-cni-binary-copy\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.440391 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440343    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-var-lib-openvswitch\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.440391 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440363    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-run-openvswitch\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.440617 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440433    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-ovnkube-script-lib\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.440617 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440459    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-kubelet-dir\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.440617 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440477    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-sys-fs\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.440617 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440498    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-modprobe-d\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.440617 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440513    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3889c85-d79f-4629-94bb-c3fe1c1471ff-system-cni-dir\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.440617 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440530    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3889c85-d79f-4629-94bb-c3fe1c1471ff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.440617 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440544    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-kubelet\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.440617 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440559    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-run-ovn-kubernetes\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.440617 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440588    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-cni-netd\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.440982 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440621    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-sysctl-d\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.440982 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440647    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-run\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.440982 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440672    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-systemd-units\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.440982 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440696    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-run-netns\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.440982 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440718    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-run-systemd\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.440982 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440742    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9aeabbb3-4723-4845-b296-bc37aa059605-agent-certs\") pod \"konnectivity-agent-bcz4n\" (UID: \"9aeabbb3-4723-4845-b296-bc37aa059605\") " pod="kube-system/konnectivity-agent-bcz4n"
Aug 04 17:59:58.440982 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440768    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-socket-dir\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.440982 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440796    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-kubernetes\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.440982 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440821    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d520936e-622a-4cf8-852f-aaa274029d02-tmp\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.440982 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440902    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rq5c\" (UniqueName: \"kubernetes.io/projected/d3889c85-d79f-4629-94bb-c3fe1c1471ff-kube-api-access-4rq5c\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.440982 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440956    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-registration-dir\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.441631 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.441045    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 04 17:59:58.441631 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.441258    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 04 17:59:58.441631 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.441273    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 04 17:59:58.441631 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.441377    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.441811 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.440984    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-device-dir\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.441811 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.441755    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8s822\""
Aug 04 17:59:58.441811 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.441769    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5wws\" (UniqueName: \"kubernetes.io/projected/952008b3-f6da-4a37-96a3-8d9b431ee939-kube-api-access-l5wws\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.441811 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.441798    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-sysctl-conf\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.441940 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.441837    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-lib-modules\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.441940 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.441862    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3889c85-d79f-4629-94bb-c3fe1c1471ff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.441940 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.441885    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-slash\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.441940 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.441909    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-node-log\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.441940 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.441933    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9aeabbb3-4723-4845-b296-bc37aa059605-konnectivity-ca\") pod \"konnectivity-agent-bcz4n\" (UID: \"9aeabbb3-4723-4845-b296-bc37aa059605\") " pod="kube-system/konnectivity-agent-bcz4n"
Aug 04 17:59:58.442157 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.441967    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-sysconfig\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.442157 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.441999    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-sys\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.442157 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442029    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d520936e-622a-4cf8-852f-aaa274029d02-etc-tuned\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.442157 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442056    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3889c85-d79f-4629-94bb-c3fe1c1471ff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.442157 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442082    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-etc-openvswitch\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.442157 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442112    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-run-ovn\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.442157 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442129    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 04 17:59:58.442520 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442160    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-log-socket\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.442520 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442191    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-cni-bin\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.442520 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442236    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-var-lib-kubelet\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.442520 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442265    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.442520 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442294    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-ovnkube-config\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.442520 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442321    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-env-overrides\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.442520 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442357    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-md8rz\""
Aug 04 17:59:58.442520 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442369    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-ovn-node-metrics-cert\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.442520 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442410    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 04 17:59:58.442520 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442397    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-etc-selinux\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.442520 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442445    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-systemd\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.442985 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.442965    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 17:59:58.443641 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:58.443260    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 17:59:58.445461 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.444024    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8pqkl\""
Aug 04 17:59:58.445461 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.444206    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 04 17:59:58.445587 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.445519    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 17:59:58.445659 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:58.445602    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 17:59:58.445725 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.445707    2543 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:58.448038 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.448020    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qnwcg"
Aug 04 17:59:58.450183 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.450160    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 17:59:58.450262 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:58.450231    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 17:59:58.450817 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.450790    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 04 17:59:58.450888 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.450816    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 04 17:59:58.450888 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.450824    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 04 17:59:58.450888 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.450822    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mpjq8\""
Aug 04 17:59:58.463176 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.463156    2543 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 17:59:58.518941 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.518891    2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 17:54:57 +0000 UTC" deadline="2028-01-24 02:56:21.418042796 +0000 UTC"
Aug 04 17:59:58.518941 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.518938    2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12896h56m22.899107677s"
Aug 04 17:59:58.536797 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.536779    2543 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 04 17:59:58.539703 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.539685    2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d7fgg"
Aug 04 17:59:58.543208 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543189    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-run-systemd\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.543325 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543220    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5wws\" (UniqueName: \"kubernetes.io/projected/952008b3-f6da-4a37-96a3-8d9b431ee939-kube-api-access-l5wws\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.543325 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543243    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 17:59:58.543325 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543267    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-kubernetes\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.543325 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543288    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-etc-kubernetes\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.543325 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543312    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-run-systemd\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.543588 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543327    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-sysctl-conf\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.543588 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543359    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-lib-modules\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.543588 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543403    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-slash\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.543588 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543408    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-kubernetes\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.543588 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543441    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-node-log\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.543588 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543445    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-sysctl-conf\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.543588 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543455    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-slash\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.543588 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543476    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-lib-modules\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.543588 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543483    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-node-log\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.543588 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543488    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-system-cni-dir\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.543588 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543516    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-multus-conf-dir\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.543588 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543542    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-sys\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.543588 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543578    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d520936e-622a-4cf8-852f-aaa274029d02-etc-tuned\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543605    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-etc-openvswitch\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543630    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-run-ovn\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543607    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-sys\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543654    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-log-socket\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543659    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-etc-openvswitch\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543678    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-cni-bin\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543703    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-log-socket\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543703    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543716    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-run-ovn\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543743    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543737    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-ovnkube-config\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543759    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-cni-bin\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543788    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-var-lib-kubelet\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543831    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-env-overrides\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543858    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-var-lib-kubelet\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543885    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-var-lib-kubelet\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543871    2543 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 04 17:59:58.544282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543886    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fa71b16a-485e-483f-b079-3c00e21b44e6-multus-daemon-config\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543938    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/59896da8-3715-4ab5-b4b7-cbce0026cc98-kubelet-config\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543967    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/59896da8-3715-4ab5-b4b7-cbce0026cc98-dbus\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.543998    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-host\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544028    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d3889c85-d79f-4629-94bb-c3fe1c1471ff-os-release\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544112    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-host\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544123    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d3889c85-d79f-4629-94bb-c3fe1c1471ff-os-release\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544147    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2tqx\" (UniqueName: \"kubernetes.io/projected/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-kube-api-access-q2tqx\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544180    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dea9d566-e866-4409-ad9a-108474118ce2-host\") pod \"node-ca-z8dv8\" (UID: \"dea9d566-e866-4409-ad9a-108474118ce2\") " pod="openshift-image-registry/node-ca-z8dv8"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544211    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcvf6\" (UniqueName: \"kubernetes.io/projected/dea9d566-e866-4409-ad9a-108474118ce2-kube-api-access-bcvf6\") pod \"node-ca-z8dv8\" (UID: \"dea9d566-e866-4409-ad9a-108474118ce2\") " pod="openshift-image-registry/node-ca-z8dv8"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544235    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-multus-cni-dir\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544255    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-env-overrides\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544260    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9gq7\" (UniqueName: \"kubernetes.io/projected/d05e7582-030c-4b50-b5ce-90e4c81c1f1e-kube-api-access-h9gq7\") pod \"node-resolver-p52sc\" (UID: \"d05e7582-030c-4b50-b5ce-90e4c81c1f1e\") " pod="openshift-dns/node-resolver-p52sc"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544290    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3889c85-d79f-4629-94bb-c3fe1c1471ff-cnibin\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544325    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-ovnkube-config\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544371    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-hostroot\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544383    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3889c85-d79f-4629-94bb-c3fe1c1471ff-cnibin\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.545204 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544407    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-modprobe-d\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544436    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3889c85-d79f-4629-94bb-c3fe1c1471ff-system-cni-dir\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544477    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3889c85-d79f-4629-94bb-c3fe1c1471ff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544501    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3889c85-d79f-4629-94bb-c3fe1c1471ff-system-cni-dir\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544507    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-kubelet\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544550    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-cni-netd\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544581    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-os-release\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544605    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-kubelet\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544608    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-sysctl-d\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544519    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-modprobe-d\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544639    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-cni-netd\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544650    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-systemd-units\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544682    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9aeabbb3-4723-4845-b296-bc37aa059605-agent-certs\") pod \"konnectivity-agent-bcz4n\" (UID: \"9aeabbb3-4723-4845-b296-bc37aa059605\") " pod="kube-system/konnectivity-agent-bcz4n"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544694    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-systemd-units\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544733    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-sysctl-d\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544737    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-socket-dir\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544764    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-registration-dir\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.546156 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544867    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-device-dir\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544892    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-registration-dir\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544908    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-run-k8s-cni-cncf-io\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.544985    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-device-dir\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545044    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3889c85-d79f-4629-94bb-c3fe1c1471ff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545076    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-socket-dir\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545103    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e118332-1ad2-42d0-b601-8c566c538432-iptables-alerter-script\") pod \"iptables-alerter-qnwcg\" (UID: \"3e118332-1ad2-42d0-b601-8c566c538432\") " pod="openshift-network-operator/iptables-alerter-qnwcg"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545140    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d520936e-622a-4cf8-852f-aaa274029d02-tmp\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545166    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rq5c\" (UniqueName: \"kubernetes.io/projected/d3889c85-d79f-4629-94bb-c3fe1c1471ff-kube-api-access-4rq5c\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545194    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa71b16a-485e-483f-b079-3c00e21b44e6-cni-binary-copy\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545221    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-multus-socket-dir-parent\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545239    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxkfg\" (UniqueName: \"kubernetes.io/projected/fa71b16a-485e-483f-b079-3c00e21b44e6-kube-api-access-gxkfg\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545255    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d05e7582-030c-4b50-b5ce-90e4c81c1f1e-hosts-file\") pod \"node-resolver-p52sc\" (UID: \"d05e7582-030c-4b50-b5ce-90e4c81c1f1e\") " pod="openshift-dns/node-resolver-p52sc"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545271    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs\") pod \"network-metrics-daemon-r27tq\" (UID: \"4fea01c7-a50c-40f7-a628-9745002f5158\") " pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545291    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3889c85-d79f-4629-94bb-c3fe1c1471ff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545317    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9aeabbb3-4723-4845-b296-bc37aa059605-konnectivity-ca\") pod \"konnectivity-agent-bcz4n\" (UID: \"9aeabbb3-4723-4845-b296-bc37aa059605\") " pod="kube-system/konnectivity-agent-bcz4n"
Aug 04 17:59:58.546997 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545338    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-var-lib-cni-bin\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.547673 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545360    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlbwx\" (UniqueName: \"kubernetes.io/projected/4fea01c7-a50c-40f7-a628-9745002f5158-kube-api-access-qlbwx\") pod \"network-metrics-daemon-r27tq\" (UID: \"4fea01c7-a50c-40f7-a628-9745002f5158\") " pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 17:59:58.547673 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545384    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e118332-1ad2-42d0-b601-8c566c538432-host-slash\") pod \"iptables-alerter-qnwcg\" (UID: \"3e118332-1ad2-42d0-b601-8c566c538432\") " pod="openshift-network-operator/iptables-alerter-qnwcg"
Aug 04 17:59:58.547673 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545426    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-sysconfig\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.547673 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545452    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3889c85-d79f-4629-94bb-c3fe1c1471ff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.547673 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545478    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-ovn-node-metrics-cert\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.547673 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545494    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-etc-selinux\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.547673 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545513    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k49qd\" (UniqueName: \"kubernetes.io/projected/3e118332-1ad2-42d0-b601-8c566c538432-kube-api-access-k49qd\") pod \"iptables-alerter-qnwcg\" (UID: \"3e118332-1ad2-42d0-b601-8c566c538432\") " pod="openshift-network-operator/iptables-alerter-qnwcg"
Aug 04 17:59:58.547673 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545530    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-run-netns\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.547673 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545546    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb5nx\" (UniqueName: \"kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx\") pod \"network-check-target-jzdgf\" (UID: \"d69a269d-b7ef-4618-aafb-475fc3de7ab2\") " pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 17:59:58.547673 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545659    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-systemd\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.547673 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545706    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-systemd\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.547673 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.545739    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-etc-sysconfig\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.547673 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.546131    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9aeabbb3-4723-4845-b296-bc37aa059605-konnectivity-ca\") pod \"konnectivity-agent-bcz4n\" (UID: \"9aeabbb3-4723-4845-b296-bc37aa059605\") " pod="kube-system/konnectivity-agent-bcz4n"
Aug 04 17:59:58.547673 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.546284    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3889c85-d79f-4629-94bb-c3fe1c1471ff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.548170 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.547681    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d520936e-622a-4cf8-852f-aaa274029d02-etc-tuned\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.548170 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.547829    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9aeabbb3-4723-4845-b296-bc37aa059605-agent-certs\") pod \"konnectivity-agent-bcz4n\" (UID: \"9aeabbb3-4723-4845-b296-bc37aa059605\") " pod="kube-system/konnectivity-agent-bcz4n"
Aug 04 17:59:58.548170 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.547938    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-etc-selinux\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.548310 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548218    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-cnibin\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.548310 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548255    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3889c85-d79f-4629-94bb-c3fe1c1471ff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.548310 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548263    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-var-lib-cni-multus\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.548310 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548298    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qwjq\" (UniqueName: \"kubernetes.io/projected/d520936e-622a-4cf8-852f-aaa274029d02-kube-api-access-4qwjq\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.548534 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548352    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3889c85-d79f-4629-94bb-c3fe1c1471ff-cni-binary-copy\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.548534 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548481    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-var-lib-openvswitch\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.548534 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548526    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-run-openvswitch\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.548697 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548581    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-ovnkube-script-lib\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.548697 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548614    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-kubelet-dir\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.548697 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548644    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-sys-fs\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.548697 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548693    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dea9d566-e866-4409-ad9a-108474118ce2-serviceca\") pod \"node-ca-z8dv8\" (UID: \"dea9d566-e866-4409-ad9a-108474118ce2\") " pod="openshift-image-registry/node-ca-z8dv8"
Aug 04 17:59:58.548902 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548722    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-run-ovn-kubernetes\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.548902 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548755    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-run-multus-certs\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.548902 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548779    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d05e7582-030c-4b50-b5ce-90e4c81c1f1e-tmp-dir\") pod \"node-resolver-p52sc\" (UID: \"d05e7582-030c-4b50-b5ce-90e4c81c1f1e\") " pod="openshift-dns/node-resolver-p52sc"
Aug 04 17:59:58.548902 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548806    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-run\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.548902 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548859    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-run-netns\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.549159 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.548956    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-run-netns\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.549159 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.549032    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3889c85-d79f-4629-94bb-c3fe1c1471ff-cni-binary-copy\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.549159 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.549036    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d520936e-622a-4cf8-852f-aaa274029d02-run\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.549159 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.549072    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-sys-fs\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.549159 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.549114    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-run-openvswitch\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.549159 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.549121    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-host-run-ovn-kubernetes\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.549159 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.549148    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-var-lib-openvswitch\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.549500 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.549269    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/952008b3-f6da-4a37-96a3-8d9b431ee939-kubelet-dir\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.549635 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.549580    2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d7fgg"
Aug 04 17:59:58.549709 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:58.549690    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c305bdf94bdf64315ff2bce273b65b.slice/crio-8536dc5f21a7b70cd71debbbdb50d6db942e004e69ab56dc8c1c3addb7c00bec WatchSource:0}: Error finding container 8536dc5f21a7b70cd71debbbdb50d6db942e004e69ab56dc8c1c3addb7c00bec: Status 404 returned error can't find the container with id 8536dc5f21a7b70cd71debbbdb50d6db942e004e69ab56dc8c1c3addb7c00bec
Aug 04 17:59:58.549970 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.549783    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-ovnkube-script-lib\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.549970 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.549945    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d520936e-622a-4cf8-852f-aaa274029d02-tmp\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.550122 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.550100    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-ovn-node-metrics-cert\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.550186 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:58.550165    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97709841e9032d4d37a1c8319191f4f.slice/crio-fea6321e0d75c75f6e3f165dafe2f1fe691db600163b0d7e9f7ebf4e66c5a81d WatchSource:0}: Error finding container fea6321e0d75c75f6e3f165dafe2f1fe691db600163b0d7e9f7ebf4e66c5a81d: Status 404 returned error can't find the container with id fea6321e0d75c75f6e3f165dafe2f1fe691db600163b0d7e9f7ebf4e66c5a81d
Aug 04 17:59:58.557380 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.557364    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 17:59:58.561918 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.561884    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5wws\" (UniqueName: \"kubernetes.io/projected/952008b3-f6da-4a37-96a3-8d9b431ee939-kube-api-access-l5wws\") pod \"aws-ebs-csi-driver-node-42jkb\" (UID: \"952008b3-f6da-4a37-96a3-8d9b431ee939\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.562085 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.562063    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qwjq\" (UniqueName: \"kubernetes.io/projected/d520936e-622a-4cf8-852f-aaa274029d02-kube-api-access-4qwjq\") pod \"tuned-hbksj\" (UID: \"d520936e-622a-4cf8-852f-aaa274029d02\") " pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.564515 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.562648    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rq5c\" (UniqueName: \"kubernetes.io/projected/d3889c85-d79f-4629-94bb-c3fe1c1471ff-kube-api-access-4rq5c\") pod \"multus-additional-cni-plugins-2zthc\" (UID: \"d3889c85-d79f-4629-94bb-c3fe1c1471ff\") " pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.565175 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.565150    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2tqx\" (UniqueName: \"kubernetes.io/projected/71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3-kube-api-access-q2tqx\") pod \"ovnkube-node-tkvl9\" (UID: \"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.574684 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.574645    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal" event={"ID":"28c305bdf94bdf64315ff2bce273b65b","Type":"ContainerStarted","Data":"8536dc5f21a7b70cd71debbbdb50d6db942e004e69ab56dc8c1c3addb7c00bec"}
Aug 04 17:59:58.575770 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.575749    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal" event={"ID":"a97709841e9032d4d37a1c8319191f4f","Type":"ContainerStarted","Data":"fea6321e0d75c75f6e3f165dafe2f1fe691db600163b0d7e9f7ebf4e66c5a81d"}
Aug 04 17:59:58.649191 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649162    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-run-k8s-cni-cncf-io\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.649282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649193    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e118332-1ad2-42d0-b601-8c566c538432-iptables-alerter-script\") pod \"iptables-alerter-qnwcg\" (UID: \"3e118332-1ad2-42d0-b601-8c566c538432\") " pod="openshift-network-operator/iptables-alerter-qnwcg"
Aug 04 17:59:58.649282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649210    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa71b16a-485e-483f-b079-3c00e21b44e6-cni-binary-copy\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.649282 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649231    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-multus-socket-dir-parent\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.649436 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649284    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-run-k8s-cni-cncf-io\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.649436 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649332    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxkfg\" (UniqueName: \"kubernetes.io/projected/fa71b16a-485e-483f-b079-3c00e21b44e6-kube-api-access-gxkfg\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.649436 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649356    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d05e7582-030c-4b50-b5ce-90e4c81c1f1e-hosts-file\") pod \"node-resolver-p52sc\" (UID: \"d05e7582-030c-4b50-b5ce-90e4c81c1f1e\") " pod="openshift-dns/node-resolver-p52sc"
Aug 04 17:59:58.649436 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649377    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs\") pod \"network-metrics-daemon-r27tq\" (UID: \"4fea01c7-a50c-40f7-a628-9745002f5158\") " pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 17:59:58.649436 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649393    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-var-lib-cni-bin\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.649436 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649407    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlbwx\" (UniqueName: \"kubernetes.io/projected/4fea01c7-a50c-40f7-a628-9745002f5158-kube-api-access-qlbwx\") pod \"network-metrics-daemon-r27tq\" (UID: \"4fea01c7-a50c-40f7-a628-9745002f5158\") " pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 17:59:58.649436 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649422    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e118332-1ad2-42d0-b601-8c566c538432-host-slash\") pod \"iptables-alerter-qnwcg\" (UID: \"3e118332-1ad2-42d0-b601-8c566c538432\") " pod="openshift-network-operator/iptables-alerter-qnwcg"
Aug 04 17:59:58.649772 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649448    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k49qd\" (UniqueName: \"kubernetes.io/projected/3e118332-1ad2-42d0-b601-8c566c538432-kube-api-access-k49qd\") pod \"iptables-alerter-qnwcg\" (UID: \"3e118332-1ad2-42d0-b601-8c566c538432\") " pod="openshift-network-operator/iptables-alerter-qnwcg"
Aug 04 17:59:58.649772 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649460    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d05e7582-030c-4b50-b5ce-90e4c81c1f1e-hosts-file\") pod \"node-resolver-p52sc\" (UID: \"d05e7582-030c-4b50-b5ce-90e4c81c1f1e\") " pod="openshift-dns/node-resolver-p52sc"
Aug 04 17:59:58.649772 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649471    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-var-lib-cni-bin\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.649772 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649525    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e118332-1ad2-42d0-b601-8c566c538432-host-slash\") pod \"iptables-alerter-qnwcg\" (UID: \"3e118332-1ad2-42d0-b601-8c566c538432\") " pod="openshift-network-operator/iptables-alerter-qnwcg"
Aug 04 17:59:58.649772 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:58.649556    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:58.649772 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649600    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-run-netns\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.649772 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:58.649650    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs podName:4fea01c7-a50c-40f7-a628-9745002f5158 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:59.149596239 +0000 UTC m=+2.120240581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs") pod "network-metrics-daemon-r27tq" (UID: "4fea01c7-a50c-40f7-a628-9745002f5158") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:58.649772 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649660    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-multus-socket-dir-parent\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.649772 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649678    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb5nx\" (UniqueName: \"kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx\") pod \"network-check-target-jzdgf\" (UID: \"d69a269d-b7ef-4618-aafb-475fc3de7ab2\") " pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 17:59:58.649772 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649648    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-run-netns\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.649772 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649716    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-cnibin\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.649772 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649742    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-var-lib-cni-multus\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.649772 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649767    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa71b16a-485e-483f-b079-3c00e21b44e6-cni-binary-copy\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649774    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-cnibin\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649772    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dea9d566-e866-4409-ad9a-108474118ce2-serviceca\") pod \"node-ca-z8dv8\" (UID: \"dea9d566-e866-4409-ad9a-108474118ce2\") " pod="openshift-image-registry/node-ca-z8dv8"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649797    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e118332-1ad2-42d0-b601-8c566c538432-iptables-alerter-script\") pod \"iptables-alerter-qnwcg\" (UID: \"3e118332-1ad2-42d0-b601-8c566c538432\") " pod="openshift-network-operator/iptables-alerter-qnwcg"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649824    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-var-lib-cni-multus\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649857    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-run-multus-certs\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649887    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-run-multus-certs\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649898    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d05e7582-030c-4b50-b5ce-90e4c81c1f1e-tmp-dir\") pod \"node-resolver-p52sc\" (UID: \"d05e7582-030c-4b50-b5ce-90e4c81c1f1e\") " pod="openshift-dns/node-resolver-p52sc"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649920    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649937    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-etc-kubernetes\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649954    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-system-cni-dir\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.649974    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-multus-conf-dir\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650004    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-etc-kubernetes\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650007    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-multus-conf-dir\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:58.650026    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650066    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-var-lib-kubelet\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650066    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-system-cni-dir\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650032    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-host-var-lib-kubelet\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650312 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:58.650087    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret podName:59896da8-3715-4ab5-b4b7-cbce0026cc98 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:59.150070451 +0000 UTC m=+2.120714771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret") pod "global-pull-secret-syncer-gsbtv" (UID: "59896da8-3715-4ab5-b4b7-cbce0026cc98") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650129    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fa71b16a-485e-483f-b079-3c00e21b44e6-multus-daemon-config\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650168    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/59896da8-3715-4ab5-b4b7-cbce0026cc98-kubelet-config\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650193    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/59896da8-3715-4ab5-b4b7-cbce0026cc98-dbus\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650221    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dea9d566-e866-4409-ad9a-108474118ce2-host\") pod \"node-ca-z8dv8\" (UID: \"dea9d566-e866-4409-ad9a-108474118ce2\") " pod="openshift-image-registry/node-ca-z8dv8"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650221    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/59896da8-3715-4ab5-b4b7-cbce0026cc98-kubelet-config\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650223    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d05e7582-030c-4b50-b5ce-90e4c81c1f1e-tmp-dir\") pod \"node-resolver-p52sc\" (UID: \"d05e7582-030c-4b50-b5ce-90e4c81c1f1e\") " pod="openshift-dns/node-resolver-p52sc"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650259    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dea9d566-e866-4409-ad9a-108474118ce2-host\") pod \"node-ca-z8dv8\" (UID: \"dea9d566-e866-4409-ad9a-108474118ce2\") " pod="openshift-image-registry/node-ca-z8dv8"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650263    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcvf6\" (UniqueName: \"kubernetes.io/projected/dea9d566-e866-4409-ad9a-108474118ce2-kube-api-access-bcvf6\") pod \"node-ca-z8dv8\" (UID: \"dea9d566-e866-4409-ad9a-108474118ce2\") " pod="openshift-image-registry/node-ca-z8dv8"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650221    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dea9d566-e866-4409-ad9a-108474118ce2-serviceca\") pod \"node-ca-z8dv8\" (UID: \"dea9d566-e866-4409-ad9a-108474118ce2\") " pod="openshift-image-registry/node-ca-z8dv8"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650281    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-multus-cni-dir\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650298    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9gq7\" (UniqueName: \"kubernetes.io/projected/d05e7582-030c-4b50-b5ce-90e4c81c1f1e-kube-api-access-h9gq7\") pod \"node-resolver-p52sc\" (UID: \"d05e7582-030c-4b50-b5ce-90e4c81c1f1e\") " pod="openshift-dns/node-resolver-p52sc"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650330    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/59896da8-3715-4ab5-b4b7-cbce0026cc98-dbus\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650334    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-hostroot\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650359    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-hostroot\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650383    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-multus-cni-dir\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650390    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-os-release\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650471    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa71b16a-485e-483f-b079-3c00e21b44e6-os-release\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.650933 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.650561    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fa71b16a-485e-483f-b079-3c00e21b44e6-multus-daemon-config\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.658427 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:58.658412    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:58.658484 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:58.658429    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:58.658484 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:58.658439    2543 projected.go:196] Error preparing data for projected volume kube-api-access-lb5nx for pod openshift-network-diagnostics/network-check-target-jzdgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:58.658484 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:58.658478    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx podName:d69a269d-b7ef-4618-aafb-475fc3de7ab2 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:59.158467866 +0000 UTC m=+2.129112207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lb5nx" (UniqueName: "kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx") pod "network-check-target-jzdgf" (UID: "d69a269d-b7ef-4618-aafb-475fc3de7ab2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:58.663699 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.663680    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcvf6\" (UniqueName: \"kubernetes.io/projected/dea9d566-e866-4409-ad9a-108474118ce2-kube-api-access-bcvf6\") pod \"node-ca-z8dv8\" (UID: \"dea9d566-e866-4409-ad9a-108474118ce2\") " pod="openshift-image-registry/node-ca-z8dv8"
Aug 04 17:59:58.663945 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.663927    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlbwx\" (UniqueName: \"kubernetes.io/projected/4fea01c7-a50c-40f7-a628-9745002f5158-kube-api-access-qlbwx\") pod \"network-metrics-daemon-r27tq\" (UID: \"4fea01c7-a50c-40f7-a628-9745002f5158\") " pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 17:59:58.664013 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.663960    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9gq7\" (UniqueName: \"kubernetes.io/projected/d05e7582-030c-4b50-b5ce-90e4c81c1f1e-kube-api-access-h9gq7\") pod \"node-resolver-p52sc\" (UID: \"d05e7582-030c-4b50-b5ce-90e4c81c1f1e\") " pod="openshift-dns/node-resolver-p52sc"
Aug 04 17:59:58.664417 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.664372    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxkfg\" (UniqueName: \"kubernetes.io/projected/fa71b16a-485e-483f-b079-3c00e21b44e6-kube-api-access-gxkfg\") pod \"multus-p4xjn\" (UID: \"fa71b16a-485e-483f-b079-3c00e21b44e6\") " pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.664490 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.664391    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k49qd\" (UniqueName: \"kubernetes.io/projected/3e118332-1ad2-42d0-b601-8c566c538432-kube-api-access-k49qd\") pod \"iptables-alerter-qnwcg\" (UID: \"3e118332-1ad2-42d0-b601-8c566c538432\") " pod="openshift-network-operator/iptables-alerter-qnwcg"
Aug 04 17:59:58.755614 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.755559    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb"
Aug 04 17:59:58.762368 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.762344    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hbksj"
Aug 04 17:59:58.762982 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:58.762958    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952008b3_f6da_4a37_96a3_8d9b431ee939.slice/crio-a9f0ad670697b258c6713f9831e58a4c163da52d9a3402c2467c604d1e70a1e2 WatchSource:0}: Error finding container a9f0ad670697b258c6713f9831e58a4c163da52d9a3402c2467c604d1e70a1e2: Status 404 returned error can't find the container with id a9f0ad670697b258c6713f9831e58a4c163da52d9a3402c2467c604d1e70a1e2
Aug 04 17:59:58.769921 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:58.769903    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd520936e_622a_4cf8_852f_aaa274029d02.slice/crio-4afd393f3cf107626a5d25663b931e4aad4a1342a44e94d51137b41ef0d5d55a WatchSource:0}: Error finding container 4afd393f3cf107626a5d25663b931e4aad4a1342a44e94d51137b41ef0d5d55a: Status 404 returned error can't find the container with id 4afd393f3cf107626a5d25663b931e4aad4a1342a44e94d51137b41ef0d5d55a
Aug 04 17:59:58.778323 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.778305    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2zthc"
Aug 04 17:59:58.785038 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:58.785018    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3889c85_d79f_4629_94bb_c3fe1c1471ff.slice/crio-cc24336982966c8abf458d1b6617ad264bc0c0187fd9e72d09a92671e4a04409 WatchSource:0}: Error finding container cc24336982966c8abf458d1b6617ad264bc0c0187fd9e72d09a92671e4a04409: Status 404 returned error can't find the container with id cc24336982966c8abf458d1b6617ad264bc0c0187fd9e72d09a92671e4a04409
Aug 04 17:59:58.794281 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.794264    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 17:59:58.802387 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:58.802346    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d9fb85_07cf_4f2a_9e77_04eaa8d80ce3.slice/crio-c8c8b2d5ad4fa6fa5790a2da05453e23b138a4f4d0732215e2ae18a873c73418 WatchSource:0}: Error finding container c8c8b2d5ad4fa6fa5790a2da05453e23b138a4f4d0732215e2ae18a873c73418: Status 404 returned error can't find the container with id c8c8b2d5ad4fa6fa5790a2da05453e23b138a4f4d0732215e2ae18a873c73418
Aug 04 17:59:58.804141 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.804126    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bcz4n"
Aug 04 17:59:58.810952 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.810934    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z8dv8"
Aug 04 17:59:58.811604 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:58.811584    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aeabbb3_4723_4845_b296_bc37aa059605.slice/crio-df96171d04e6ebe7b5e88d7cb8f02833ce258c0e696d4b3bdd1487d3150a8574 WatchSource:0}: Error finding container df96171d04e6ebe7b5e88d7cb8f02833ce258c0e696d4b3bdd1487d3150a8574: Status 404 returned error can't find the container with id df96171d04e6ebe7b5e88d7cb8f02833ce258c0e696d4b3bdd1487d3150a8574
Aug 04 17:59:58.817791 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:58.817770    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea9d566_e866_4409_ad9a_108474118ce2.slice/crio-27a6c94e8ded4fc028e933f5c9344f397424078b2252550ce8e1ec231f14900b WatchSource:0}: Error finding container 27a6c94e8ded4fc028e933f5c9344f397424078b2252550ce8e1ec231f14900b: Status 404 returned error can't find the container with id 27a6c94e8ded4fc028e933f5c9344f397424078b2252550ce8e1ec231f14900b
Aug 04 17:59:58.830897 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.830878    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p52sc"
Aug 04 17:59:58.836560 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.836544    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p4xjn"
Aug 04 17:59:58.836974 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:58.836956    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05e7582_030c_4b50_b5ce_90e4c81c1f1e.slice/crio-8f9f84115e684a8c29d20e124a41132e8c4a38a85e8299aa0ce937c5a60a4699 WatchSource:0}: Error finding container 8f9f84115e684a8c29d20e124a41132e8c4a38a85e8299aa0ce937c5a60a4699: Status 404 returned error can't find the container with id 8f9f84115e684a8c29d20e124a41132e8c4a38a85e8299aa0ce937c5a60a4699
Aug 04 17:59:58.844242 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:58.844223    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa71b16a_485e_483f_b079_3c00e21b44e6.slice/crio-d8aba9549a8d1060be75b3bf0b2de9b2dc012dce7110fdba78e3506cdfe34e0d WatchSource:0}: Error finding container d8aba9549a8d1060be75b3bf0b2de9b2dc012dce7110fdba78e3506cdfe34e0d: Status 404 returned error can't find the container with id d8aba9549a8d1060be75b3bf0b2de9b2dc012dce7110fdba78e3506cdfe34e0d
Aug 04 17:59:58.845799 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:58.845780    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qnwcg"
Aug 04 17:59:58.853836 ip-10-0-139-237 kubenswrapper[2543]: W0804 17:59:58.853816    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e118332_1ad2_42d0_b601_8c566c538432.slice/crio-217b4bd0ffa06c2db561de85a280ea90fe3a2ebff73f10d50b34c3b0cc3ac766 WatchSource:0}: Error finding container 217b4bd0ffa06c2db561de85a280ea90fe3a2ebff73f10d50b34c3b0cc3ac766: Status 404 returned error can't find the container with id 217b4bd0ffa06c2db561de85a280ea90fe3a2ebff73f10d50b34c3b0cc3ac766
Aug 04 17:59:59.154497 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.154400    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs\") pod \"network-metrics-daemon-r27tq\" (UID: \"4fea01c7-a50c-40f7-a628-9745002f5158\") " pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 17:59:59.154497 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.154479    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 17:59:59.154702 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:59.154618    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:59.154702 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:59.154674    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret podName:59896da8-3715-4ab5-b4b7-cbce0026cc98 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:00.154660643 +0000 UTC m=+3.125304963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret") pod "global-pull-secret-syncer-gsbtv" (UID: "59896da8-3715-4ab5-b4b7-cbce0026cc98") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:59.154702 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:59.154618    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:59.154882 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:59.154761    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs podName:4fea01c7-a50c-40f7-a628-9745002f5158 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:00.15473453 +0000 UTC m=+3.125378850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs") pod "network-metrics-daemon-r27tq" (UID: "4fea01c7-a50c-40f7-a628-9745002f5158") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:59.256287 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.255393    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb5nx\" (UniqueName: \"kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx\") pod \"network-check-target-jzdgf\" (UID: \"d69a269d-b7ef-4618-aafb-475fc3de7ab2\") " pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 17:59:59.256287 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:59.255615    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:59.256287 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:59.255637    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:59.256287 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:59.255650    2543 projected.go:196] Error preparing data for projected volume kube-api-access-lb5nx for pod openshift-network-diagnostics/network-check-target-jzdgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:59.256287 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:59.255704    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx podName:d69a269d-b7ef-4618-aafb-475fc3de7ab2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:00.255686513 +0000 UTC m=+3.226330849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lb5nx" (UniqueName: "kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx") pod "network-check-target-jzdgf" (UID: "d69a269d-b7ef-4618-aafb-475fc3de7ab2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:59.553659 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.553478    2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:54:58 +0000 UTC" deadline="2028-05-12 08:28:34.688420756 +0000 UTC"
Aug 04 17:59:59.553659 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.553547    2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15518h28m35.134877764s"
Aug 04 17:59:59.572846 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.571904    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 17:59:59.572846 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:59.572034    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 17:59:59.572846 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.572435    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 17:59:59.572846 ip-10-0-139-237 kubenswrapper[2543]: E0804 17:59:59.572525    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 17:59:59.599584 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.599513    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qnwcg" event={"ID":"3e118332-1ad2-42d0-b601-8c566c538432","Type":"ContainerStarted","Data":"217b4bd0ffa06c2db561de85a280ea90fe3a2ebff73f10d50b34c3b0cc3ac766"}
Aug 04 17:59:59.615126 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.615097    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p52sc" event={"ID":"d05e7582-030c-4b50-b5ce-90e4c81c1f1e","Type":"ContainerStarted","Data":"8f9f84115e684a8c29d20e124a41132e8c4a38a85e8299aa0ce937c5a60a4699"}
Aug 04 17:59:59.630789 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.630761    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bcz4n" event={"ID":"9aeabbb3-4723-4845-b296-bc37aa059605","Type":"ContainerStarted","Data":"df96171d04e6ebe7b5e88d7cb8f02833ce258c0e696d4b3bdd1487d3150a8574"}
Aug 04 17:59:59.648587 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.648562    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hbksj" event={"ID":"d520936e-622a-4cf8-852f-aaa274029d02","Type":"ContainerStarted","Data":"4afd393f3cf107626a5d25663b931e4aad4a1342a44e94d51137b41ef0d5d55a"}
Aug 04 17:59:59.661698 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.661671    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p4xjn" event={"ID":"fa71b16a-485e-483f-b079-3c00e21b44e6","Type":"ContainerStarted","Data":"d8aba9549a8d1060be75b3bf0b2de9b2dc012dce7110fdba78e3506cdfe34e0d"}
Aug 04 17:59:59.670370 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.670306    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z8dv8" event={"ID":"dea9d566-e866-4409-ad9a-108474118ce2","Type":"ContainerStarted","Data":"27a6c94e8ded4fc028e933f5c9344f397424078b2252550ce8e1ec231f14900b"}
Aug 04 17:59:59.676039 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.676015    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9" event={"ID":"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3","Type":"ContainerStarted","Data":"c8c8b2d5ad4fa6fa5790a2da05453e23b138a4f4d0732215e2ae18a873c73418"}
Aug 04 17:59:59.678924 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.678896    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zthc" event={"ID":"d3889c85-d79f-4629-94bb-c3fe1c1471ff","Type":"ContainerStarted","Data":"cc24336982966c8abf458d1b6617ad264bc0c0187fd9e72d09a92671e4a04409"}
Aug 04 17:59:59.691485 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.691458    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb" event={"ID":"952008b3-f6da-4a37-96a3-8d9b431ee939","Type":"ContainerStarted","Data":"a9f0ad670697b258c6713f9831e58a4c163da52d9a3402c2467c604d1e70a1e2"}
Aug 04 17:59:59.803464 ip-10-0-139-237 kubenswrapper[2543]: I0804 17:59:59.803437    2543 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:00.162868 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:00.162076    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs\") pod \"network-metrics-daemon-r27tq\" (UID: \"4fea01c7-a50c-40f7-a628-9745002f5158\") " pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:00.162868 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:00.162150    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:00.162868 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:00.162281    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:00.162868 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:00.162361    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret podName:59896da8-3715-4ab5-b4b7-cbce0026cc98 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:02.162323659 +0000 UTC m=+5.132967982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret") pod "global-pull-secret-syncer-gsbtv" (UID: "59896da8-3715-4ab5-b4b7-cbce0026cc98") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:00.162868 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:00.162768    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:00.162868 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:00.162820    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs podName:4fea01c7-a50c-40f7-a628-9745002f5158 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:02.162804822 +0000 UTC m=+5.133449147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs") pod "network-metrics-daemon-r27tq" (UID: "4fea01c7-a50c-40f7-a628-9745002f5158") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:00.258792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:00.258763    2543 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:00.262619 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:00.262582    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb5nx\" (UniqueName: \"kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx\") pod \"network-check-target-jzdgf\" (UID: \"d69a269d-b7ef-4618-aafb-475fc3de7ab2\") " pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:00.262746 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:00.262724    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:00.262746 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:00.262742    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:00.262862 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:00.262756    2543 projected.go:196] Error preparing data for projected volume kube-api-access-lb5nx for pod openshift-network-diagnostics/network-check-target-jzdgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:00.262862 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:00.262806    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx podName:d69a269d-b7ef-4618-aafb-475fc3de7ab2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:02.262789191 +0000 UTC m=+5.233433511 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lb5nx" (UniqueName: "kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx") pod "network-check-target-jzdgf" (UID: "d69a269d-b7ef-4618-aafb-475fc3de7ab2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:00.553742 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:00.553656    2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:54:58 +0000 UTC" deadline="2028-01-20 00:52:13.087914809 +0000 UTC"
Aug 04 18:00:00.553742 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:00.553692    2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12798h52m12.534226875s"
Aug 04 18:00:00.569614 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:00.569590    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:00.569776 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:00.569715    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:00.718618 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:00.718580    2543 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:01.572746 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:01.572499    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:01.572746 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:01.572690    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:01.573362 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:01.573083    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:01.573362 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:01.573183    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:02.179037 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:02.178999    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs\") pod \"network-metrics-daemon-r27tq\" (UID: \"4fea01c7-a50c-40f7-a628-9745002f5158\") " pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:02.179227 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:02.179076    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:02.179227 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:02.179213    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:02.179361 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:02.179275    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret podName:59896da8-3715-4ab5-b4b7-cbce0026cc98 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:06.179256984 +0000 UTC m=+9.149901318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret") pod "global-pull-secret-syncer-gsbtv" (UID: "59896da8-3715-4ab5-b4b7-cbce0026cc98") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:02.179786 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:02.179683    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:02.179786 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:02.179740    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs podName:4fea01c7-a50c-40f7-a628-9745002f5158 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:06.179725036 +0000 UTC m=+9.150369361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs") pod "network-metrics-daemon-r27tq" (UID: "4fea01c7-a50c-40f7-a628-9745002f5158") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:02.280315 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:02.279687    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb5nx\" (UniqueName: \"kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx\") pod \"network-check-target-jzdgf\" (UID: \"d69a269d-b7ef-4618-aafb-475fc3de7ab2\") " pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:02.280315 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:02.279861    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:02.280315 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:02.279883    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:02.280315 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:02.279896    2543 projected.go:196] Error preparing data for projected volume kube-api-access-lb5nx for pod openshift-network-diagnostics/network-check-target-jzdgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:02.280315 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:02.279960    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx podName:d69a269d-b7ef-4618-aafb-475fc3de7ab2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:06.279942648 +0000 UTC m=+9.250586992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lb5nx" (UniqueName: "kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx") pod "network-check-target-jzdgf" (UID: "d69a269d-b7ef-4618-aafb-475fc3de7ab2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:02.569875 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:02.569387    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:02.569875 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:02.569517    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:03.569637 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:03.569581    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:03.570105 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:03.569787    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:03.570105 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:03.570014    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:03.570241 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:03.570105    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:04.569698 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:04.569663    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:04.570128 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:04.569803    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:05.569214 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:05.569174    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:05.569413 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:05.569297    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:05.569778 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:05.569751    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:05.570160 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:05.569876    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:06.211936 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:06.210888    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs\") pod \"network-metrics-daemon-r27tq\" (UID: \"4fea01c7-a50c-40f7-a628-9745002f5158\") " pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:06.211936 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:06.210965    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:06.211936 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:06.211088    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:06.211936 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:06.211144    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret podName:59896da8-3715-4ab5-b4b7-cbce0026cc98 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:14.211127427 +0000 UTC m=+17.181771762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret") pod "global-pull-secret-syncer-gsbtv" (UID: "59896da8-3715-4ab5-b4b7-cbce0026cc98") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:06.211936 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:06.211550    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:06.211936 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:06.211601    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs podName:4fea01c7-a50c-40f7-a628-9745002f5158 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:14.211585491 +0000 UTC m=+17.182229813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs") pod "network-metrics-daemon-r27tq" (UID: "4fea01c7-a50c-40f7-a628-9745002f5158") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:06.312254 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:06.312136    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb5nx\" (UniqueName: \"kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx\") pod \"network-check-target-jzdgf\" (UID: \"d69a269d-b7ef-4618-aafb-475fc3de7ab2\") " pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:06.312406 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:06.312295    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:06.312406 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:06.312327    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:06.312406 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:06.312337    2543 projected.go:196] Error preparing data for projected volume kube-api-access-lb5nx for pod openshift-network-diagnostics/network-check-target-jzdgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:06.312406 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:06.312384    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx podName:d69a269d-b7ef-4618-aafb-475fc3de7ab2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:14.31236853 +0000 UTC m=+17.283012861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lb5nx" (UniqueName: "kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx") pod "network-check-target-jzdgf" (UID: "d69a269d-b7ef-4618-aafb-475fc3de7ab2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:06.569677 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:06.569594    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:06.569853 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:06.569718    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:07.570367 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:07.570338    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:07.570805 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:07.570455    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:07.570805 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:07.570501    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:07.570805 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:07.570553    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:08.569441 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:08.569405    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:08.569612 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:08.569511    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:09.569936 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:09.569906    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:09.570378 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:09.570044    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:09.570378 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:09.570171    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:09.570378 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:09.570258    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:10.569155 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:10.569127    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:10.569334 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:10.569236    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:11.569253 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:11.569217    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:11.569700 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:11.569358    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:11.569700 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:11.569441    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:11.569700 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:11.569526    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:12.569923 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:12.569884    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:12.570357 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:12.570003    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:13.569452 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:13.569412    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:13.569627 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:13.569417    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:13.569627 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:13.569540    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:13.569730 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:13.569620    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:14.270825 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:14.270789    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs\") pod \"network-metrics-daemon-r27tq\" (UID: \"4fea01c7-a50c-40f7-a628-9745002f5158\") " pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:14.271270 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:14.270845    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:14.271270 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:14.270926    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:14.271270 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:14.270942    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:14.271270 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:14.270978    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret podName:59896da8-3715-4ab5-b4b7-cbce0026cc98 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:30.270962315 +0000 UTC m=+33.241606634 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret") pod "global-pull-secret-syncer-gsbtv" (UID: "59896da8-3715-4ab5-b4b7-cbce0026cc98") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:14.271270 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:14.271004    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs podName:4fea01c7-a50c-40f7-a628-9745002f5158 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:30.270985947 +0000 UTC m=+33.241630291 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs") pod "network-metrics-daemon-r27tq" (UID: "4fea01c7-a50c-40f7-a628-9745002f5158") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:14.371895 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:14.371863    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb5nx\" (UniqueName: \"kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx\") pod \"network-check-target-jzdgf\" (UID: \"d69a269d-b7ef-4618-aafb-475fc3de7ab2\") " pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:14.372043 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:14.372008    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:14.372043 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:14.372022    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:14.372043 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:14.372031    2543 projected.go:196] Error preparing data for projected volume kube-api-access-lb5nx for pod openshift-network-diagnostics/network-check-target-jzdgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:14.372208 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:14.372074    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx podName:d69a269d-b7ef-4618-aafb-475fc3de7ab2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:30.372060079 +0000 UTC m=+33.342704399 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lb5nx" (UniqueName: "kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx") pod "network-check-target-jzdgf" (UID: "d69a269d-b7ef-4618-aafb-475fc3de7ab2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:14.569528 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:14.569453    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:14.569689 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:14.569584    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:15.569524 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:15.569496    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:15.569940 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:15.569667    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:15.569940 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:15.569735    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:15.569940 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:15.569815    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:16.569106 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:16.569076    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:16.569261 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:16.569190    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:17.570185 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:17.569999    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:17.570616 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:17.570064    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:17.570616 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:17.570269    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:17.570616 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:17.570344    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:18.569063 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:18.569041    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:18.569167 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:18.569147    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:18.730500 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:18.730468    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hbksj" event={"ID":"d520936e-622a-4cf8-852f-aaa274029d02","Type":"ContainerStarted","Data":"95d13d84bdee2251190b43972b434d0921145a7a2d8ff6f092b1ba92c84232cb"}
Aug 04 18:00:18.731800 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:18.731772    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal" event={"ID":"28c305bdf94bdf64315ff2bce273b65b","Type":"ContainerStarted","Data":"ff2cd18b40005eeda90b59406a87558feb5a326de7933b59e9973eebfbabc1ab"}
Aug 04 18:00:18.733934 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:18.733910    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p4xjn" event={"ID":"fa71b16a-485e-483f-b079-3c00e21b44e6","Type":"ContainerStarted","Data":"ebaed04deb7b8628fdc844ea997f6b37c0f251bbf7bb3991a1cc2909463bb6cb"}
Aug 04 18:00:18.737431 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:18.737410    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9" event={"ID":"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3","Type":"ContainerStarted","Data":"10f97af321fea8ffeb63634f49bf0d1c3c5117a90fc810d8b0896037e43b60e1"}
Aug 04 18:00:18.737541 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:18.737465    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9" event={"ID":"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3","Type":"ContainerStarted","Data":"6ca723e5e67ca1fb2b2461c857dc4fe6ec70a11889297f77536075f9ca07aaf5"}
Aug 04 18:00:18.737541 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:18.737500    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9" event={"ID":"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3","Type":"ContainerStarted","Data":"e23f4c86cc5ccdb088f5eb81920b04cb4f5c2b95b3cd2223f7a5f78c0da438b6"}
Aug 04 18:00:18.737541 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:18.737528    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9" event={"ID":"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3","Type":"ContainerStarted","Data":"6174de48539a8870c87b36a1fcb0206bcbda8ff67cbdf1afd8b1dac8ef634188"}
Aug 04 18:00:18.737650 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:18.737561    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9" event={"ID":"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3","Type":"ContainerStarted","Data":"1af4460e8ff6acc27d1555903cf7a8726ee148a98922bdb0cdb63be839dc1f4e"}
Aug 04 18:00:18.737650 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:18.737593    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9" event={"ID":"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3","Type":"ContainerStarted","Data":"dcc4ebf91ce1467dea0748bd1f5e3b3d08f83ff33e75f7d41e6ff13c0500b4af"}
Aug 04 18:00:18.749732 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:18.749693    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hbksj" podStartSLOduration=2.695116016 podStartE2EDuration="21.749680497s" totalImagesPullingTime="19.054564481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:18.749506921 +0000 UTC m=+21.720151263" watchObservedRunningTime="2026-08-04 18:00:18.749680497 +0000 UTC m=+21.720324835"
Aug 04 18:00:18.768298 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:18.768263    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p4xjn" podStartSLOduration=1.757898097 podStartE2EDuration="20.76825163s" totalImagesPullingTime="19.010353533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:18.767975388 +0000 UTC m=+21.738619729" watchObservedRunningTime="2026-08-04 18:00:18.76825163 +0000 UTC m=+21.738895969"
Aug 04 18:00:18.786033 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:18.785966    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal" podStartSLOduration=20.785864058 podStartE2EDuration="20.785864058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:18.785673765 +0000 UTC m=+21.756318108" watchObservedRunningTime="2026-08-04 18:00:18.785864058 +0000 UTC m=+21.756508399"
Aug 04 18:00:19.569096 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:19.569071    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:19.569223 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:19.569202    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:19.569273 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:19.569253    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:19.569398 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:19.569378    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:19.741341 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:19.741289    2543 generic.go:356] "Generic (PLEG): container finished" podID="a97709841e9032d4d37a1c8319191f4f" containerID="3e666483ebe7ad7199f3c103d06d4ae7fc768d5d82f2059c6af0184262f373b1" exitCode=0
Aug 04 18:00:19.741929 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:19.741350    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal" event={"ID":"a97709841e9032d4d37a1c8319191f4f","Type":"ContainerDied","Data":"3e666483ebe7ad7199f3c103d06d4ae7fc768d5d82f2059c6af0184262f373b1"}
Aug 04 18:00:19.742987 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:19.742963    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z8dv8" event={"ID":"dea9d566-e866-4409-ad9a-108474118ce2","Type":"ContainerStarted","Data":"0ef780b85065d97d3443aa5682462656834f9109d91335b0e0370f2297a63837"}
Aug 04 18:00:19.744607 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:19.744579    2543 generic.go:356] "Generic (PLEG): container finished" podID="d3889c85-d79f-4629-94bb-c3fe1c1471ff" containerID="62604d3a09b8141147b12b892544918ac9b46686ef1178a5c09dff1ca3fc37a5" exitCode=0
Aug 04 18:00:19.744696 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:19.744670    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zthc" event={"ID":"d3889c85-d79f-4629-94bb-c3fe1c1471ff","Type":"ContainerDied","Data":"62604d3a09b8141147b12b892544918ac9b46686ef1178a5c09dff1ca3fc37a5"}
Aug 04 18:00:19.746505 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:19.746480    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb" event={"ID":"952008b3-f6da-4a37-96a3-8d9b431ee939","Type":"ContainerStarted","Data":"55c3ce4d70299bdf94c5cbb5591b687c8acbf957b81efeb29291beb67cfb754b"}
Aug 04 18:00:19.747984 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:19.747957    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qnwcg" event={"ID":"3e118332-1ad2-42d0-b601-8c566c538432","Type":"ContainerStarted","Data":"8495ee5b96f14e0b41c68ee1919b6ca6929b75ccb103fd708d8f359ffb560141"}
Aug 04 18:00:19.749551 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:19.749531    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p52sc" event={"ID":"d05e7582-030c-4b50-b5ce-90e4c81c1f1e","Type":"ContainerStarted","Data":"73e8fe90bc775cedcdad31af61b67c86b0cf2c26566fc71254b6e48c63bd32fe"}
Aug 04 18:00:19.751001 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:19.750977    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bcz4n" event={"ID":"9aeabbb3-4723-4845-b296-bc37aa059605","Type":"ContainerStarted","Data":"fe8a79e9cb5f5fdf50718fd06dd6addabcc2403daf5a885b16a99ca66de797d8"}
Aug 04 18:00:19.776237 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:19.776196    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z8dv8" podStartSLOduration=2.795042032 podStartE2EDuration="21.77618062s" totalImagesPullingTime="18.981138588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:19.775672357 +0000 UTC m=+22.746316699" watchObservedRunningTime="2026-08-04 18:00:19.77618062 +0000 UTC m=+22.746824962"
Aug 04 18:00:19.824650 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:19.824398    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bcz4n" podStartSLOduration=2.837268103 podStartE2EDuration="21.824383898s" totalImagesPullingTime="18.987115795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:19.799500378 +0000 UTC m=+22.770144717" watchObservedRunningTime="2026-08-04 18:00:19.824383898 +0000 UTC m=+22.795028238"
Aug 04 18:00:19.846258 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:19.846226    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qnwcg" podStartSLOduration=2.903856439 podStartE2EDuration="21.846215774s" totalImagesPullingTime="18.942359335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:19.846099652 +0000 UTC m=+22.816743992" watchObservedRunningTime="2026-08-04 18:00:19.846215774 +0000 UTC m=+22.816860113"
Aug 04 18:00:19.846371 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:19.846340    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-p52sc" podStartSLOduration=2.886193833 podStartE2EDuration="21.84633479s" totalImagesPullingTime="18.960140957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:19.8246782 +0000 UTC m=+22.795322535" watchObservedRunningTime="2026-08-04 18:00:19.84633479 +0000 UTC m=+22.816979128"
Aug 04 18:00:20.288215 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:20.288175    2543 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 04 18:00:20.508375 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:20.508233    2543 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T18:00:20.288195458Z","UUID":"9bfec1fd-62ed-471a-ba60-a244877fae7a","Handler":null,"Name":"","Endpoint":""}
Aug 04 18:00:20.509649 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:20.509632    2543 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 04 18:00:20.509758 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:20.509668    2543 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 04 18:00:20.569552 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:20.569530    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:20.569649 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:20.569635    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:20.755533 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:20.755502    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal" event={"ID":"a97709841e9032d4d37a1c8319191f4f","Type":"ContainerStarted","Data":"ac770f4ded2947b74417673a46a8aca886614aad756d55cd769fd5660860e4f4"}
Aug 04 18:00:20.757506 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:20.757482    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb" event={"ID":"952008b3-f6da-4a37-96a3-8d9b431ee939","Type":"ContainerStarted","Data":"99c139561cb500e57041e88b9d5783307f70a1e93e3c8d3285447be3f4b4d0dc"}
Aug 04 18:00:20.774275 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:20.774190    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal" podStartSLOduration=22.774172189 podStartE2EDuration="22.774172189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:20.773729235 +0000 UTC m=+23.744373575" watchObservedRunningTime="2026-08-04 18:00:20.774172189 +0000 UTC m=+23.744816529"
Aug 04 18:00:20.822813 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:20.822787    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bcz4n"
Aug 04 18:00:20.823646 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:20.823494    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bcz4n"
Aug 04 18:00:21.569628 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:21.569585    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:21.569628 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:21.569619    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:21.570092 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:21.569716    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:21.570092 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:21.569862    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:21.762509 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:21.762471    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb" event={"ID":"952008b3-f6da-4a37-96a3-8d9b431ee939","Type":"ContainerStarted","Data":"469c223fb980383ba354c5d09d8520088471b0ea26a876844667295d233597d0"}
Aug 04 18:00:21.766349 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:21.766292    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9" event={"ID":"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3","Type":"ContainerStarted","Data":"39333db230e488647166073dc268ca15ea29856f42072631e522aca137ca768e"}
Aug 04 18:00:21.781774 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:21.781726    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jkb" podStartSLOduration=3.485421559 podStartE2EDuration="24.781709592s" totalImagesPullingTime="21.296288033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:21.781027447 +0000 UTC m=+24.751671786" watchObservedRunningTime="2026-08-04 18:00:21.781709592 +0000 UTC m=+24.752353933"
Aug 04 18:00:21.854515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:21.854464    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bcz4n"
Aug 04 18:00:21.855145 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:21.855127    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bcz4n"
Aug 04 18:00:22.568923 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:22.568896    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:22.569086 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:22.569013    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:23.569245 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:23.569216    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:23.569656 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:23.569402    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:23.569656 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:23.569456    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:23.569656 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:23.569564    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:24.569262 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:24.569088    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:24.569808 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:24.569329    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:24.776048 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:24.776025    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9" event={"ID":"71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3","Type":"ContainerStarted","Data":"3e507396f443ecd708738cafbc1b088b8c2f5a065fae11d86fc53c06929e867d"}
Aug 04 18:00:24.776247 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:24.776238    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 18:00:24.776325 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:24.776299    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 18:00:24.777985 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:24.777961    2543 generic.go:356] "Generic (PLEG): container finished" podID="d3889c85-d79f-4629-94bb-c3fe1c1471ff" containerID="6193293f86df66e18c7c16d974dd2293f8581e2c4f618d6d82aa0a217f4e42c8" exitCode=0
Aug 04 18:00:24.778074 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:24.778005    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zthc" event={"ID":"d3889c85-d79f-4629-94bb-c3fe1c1471ff","Type":"ContainerDied","Data":"6193293f86df66e18c7c16d974dd2293f8581e2c4f618d6d82aa0a217f4e42c8"}
Aug 04 18:00:24.790382 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:24.790366    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 18:00:24.808030 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:24.807886    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9" podStartSLOduration=7.565827938 podStartE2EDuration="26.807813228s" totalImagesPullingTime="19.24198529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:24.806390111 +0000 UTC m=+27.777034445" watchObservedRunningTime="2026-08-04 18:00:24.807813228 +0000 UTC m=+27.778457571"
Aug 04 18:00:25.569197 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:25.569131    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:25.569197 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:25.569188    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:25.569722 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:25.569302    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:25.569722 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:25.569419    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:25.780449 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:25.780428    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 18:00:25.784698 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:25.784676    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jzdgf"]
Aug 04 18:00:25.784806 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:25.784795    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:25.784908 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:25.784883    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:25.788465 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:25.788447    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r27tq"]
Aug 04 18:00:25.788534 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:25.788521    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:25.788609 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:25.788594    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:25.790644 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:25.790621    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gsbtv"]
Aug 04 18:00:25.790742 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:25.790721    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:25.790846 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:25.790827    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:25.796018 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:25.796003    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 18:00:26.783580 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:26.783385    2543 generic.go:356] "Generic (PLEG): container finished" podID="d3889c85-d79f-4629-94bb-c3fe1c1471ff" containerID="932ed5cd4d04e5fe856f57e961caa6f839afec01c0319036b2fc6352d97bb9ac" exitCode=0
Aug 04 18:00:26.783882 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:26.783431    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zthc" event={"ID":"d3889c85-d79f-4629-94bb-c3fe1c1471ff","Type":"ContainerDied","Data":"932ed5cd4d04e5fe856f57e961caa6f839afec01c0319036b2fc6352d97bb9ac"}
Aug 04 18:00:27.570407 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:27.570377    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:27.570575 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:27.570469    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:27.570575 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:27.570479    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:27.570575 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:27.570501    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:27.570729 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:27.570596    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:27.570729 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:27.570683    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:28.789041 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:28.789012    2543 generic.go:356] "Generic (PLEG): container finished" podID="d3889c85-d79f-4629-94bb-c3fe1c1471ff" containerID="b8eb899ed11c8b34152168dc3d88f41d0d0e84d71f0c2e92352811eeea565e5a" exitCode=0
Aug 04 18:00:28.789356 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:28.789061    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zthc" event={"ID":"d3889c85-d79f-4629-94bb-c3fe1c1471ff","Type":"ContainerDied","Data":"b8eb899ed11c8b34152168dc3d88f41d0d0e84d71f0c2e92352811eeea565e5a"}
Aug 04 18:00:29.569426 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:29.569369    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:29.569532 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:29.569461    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzdgf" podUID="d69a269d-b7ef-4618-aafb-475fc3de7ab2"
Aug 04 18:00:29.569532 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:29.569370    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:29.569532 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:29.569527    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r27tq" podUID="4fea01c7-a50c-40f7-a628-9745002f5158"
Aug 04 18:00:29.569647 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:29.569370    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:29.569647 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:29.569582    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gsbtv" podUID="59896da8-3715-4ab5-b4b7-cbce0026cc98"
Aug 04 18:00:30.298590 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.298566    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:30.298933 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.298604    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs\") pod \"network-metrics-daemon-r27tq\" (UID: \"4fea01c7-a50c-40f7-a628-9745002f5158\") " pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:30.298933 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:30.298686    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:30.298933 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:30.298714    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:30.298933 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:30.298727    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs podName:4fea01c7-a50c-40f7-a628-9745002f5158 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:02.298714094 +0000 UTC m=+65.269358412 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs") pod "network-metrics-daemon-r27tq" (UID: "4fea01c7-a50c-40f7-a628-9745002f5158") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:30.298933 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:30.298769    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret podName:59896da8-3715-4ab5-b4b7-cbce0026cc98 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:02.298751572 +0000 UTC m=+65.269395895 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret") pod "global-pull-secret-syncer-gsbtv" (UID: "59896da8-3715-4ab5-b4b7-cbce0026cc98") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:30.399538 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.399516    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb5nx\" (UniqueName: \"kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx\") pod \"network-check-target-jzdgf\" (UID: \"d69a269d-b7ef-4618-aafb-475fc3de7ab2\") " pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:30.399632 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:30.399604    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:30.399632 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:30.399615    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:30.399632 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:30.399624    2543 projected.go:196] Error preparing data for projected volume kube-api-access-lb5nx for pod openshift-network-diagnostics/network-check-target-jzdgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:30.399726 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:30.399655    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx podName:d69a269d-b7ef-4618-aafb-475fc3de7ab2 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:02.399646897 +0000 UTC m=+65.370291215 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lb5nx" (UniqueName: "kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx") pod "network-check-target-jzdgf" (UID: "d69a269d-b7ef-4618-aafb-475fc3de7ab2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:30.816730 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.816710    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeReady"
Aug 04 18:00:30.816868 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.816785    2543 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 04 18:00:30.857368 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.857344    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54954fb884-m76mm"]
Aug 04 18:00:30.859216 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.859200    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:30.861769 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.861583    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Aug 04 18:00:30.861864 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.861690    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5b4qm\""
Aug 04 18:00:30.861864 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.861786    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Aug 04 18:00:30.861950 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.861919    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Aug 04 18:00:30.870071 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.870044    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Aug 04 18:00:30.876235 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.876213    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fk7ll"]
Aug 04 18:00:30.878948 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.878926    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r9ntf"]
Aug 04 18:00:30.879028 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.878989    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fk7ll"
Aug 04 18:00:30.881173 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.881154    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54954fb884-m76mm"]
Aug 04 18:00:30.881264 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.881224    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:30.881818 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.881797    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 04 18:00:30.881925 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.881828    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 04 18:00:30.881925 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.881801    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 04 18:00:30.882070 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.882056    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rzh2g\""
Aug 04 18:00:30.883782 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.883762    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 04 18:00:30.883782 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.883779    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k2cqb\""
Aug 04 18:00:30.883897 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.883811    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 04 18:00:30.890615 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.890569    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r9ntf"]
Aug 04 18:00:30.893704 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:30.893687    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fk7ll"]
Aug 04 18:00:31.003041 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.003020    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-image-registry-private-configuration\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.003112 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.003044    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-ca-trust-extracted\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.003112 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.003062    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-certificates\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.003112 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.003080    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-bound-sa-token\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.003112 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.003102    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62c75b76-1fdb-4555-a35e-78d0d361c064-tmp-dir\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:31.003238 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.003124    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbrtb\" (UniqueName: \"kubernetes.io/projected/62c75b76-1fdb-4555-a35e-78d0d361c064-kube-api-access-tbrtb\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:31.003238 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.003175    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62c75b76-1fdb-4555-a35e-78d0d361c064-config-volume\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:31.003238 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.003196    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:31.003238 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.003216    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-trusted-ca\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.003238 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.003231    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtzdg\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-kube-api-access-dtzdg\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.003400 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.003301    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-installation-pull-secrets\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.003400 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.003336    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68ff79ec-393a-456b-9041-3389e96656fc-cert\") pod \"ingress-canary-fk7ll\" (UID: \"68ff79ec-393a-456b-9041-3389e96656fc\") " pod="openshift-ingress-canary/ingress-canary-fk7ll"
Aug 04 18:00:31.003400 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.003356    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpfbk\" (UniqueName: \"kubernetes.io/projected/68ff79ec-393a-456b-9041-3389e96656fc-kube-api-access-rpfbk\") pod \"ingress-canary-fk7ll\" (UID: \"68ff79ec-393a-456b-9041-3389e96656fc\") " pod="openshift-ingress-canary/ingress-canary-fk7ll"
Aug 04 18:00:31.003400 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.003383    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.104294 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.104276    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-installation-pull-secrets\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.104390 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.104297    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68ff79ec-393a-456b-9041-3389e96656fc-cert\") pod \"ingress-canary-fk7ll\" (UID: \"68ff79ec-393a-456b-9041-3389e96656fc\") " pod="openshift-ingress-canary/ingress-canary-fk7ll"
Aug 04 18:00:31.104390 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.104335    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpfbk\" (UniqueName: \"kubernetes.io/projected/68ff79ec-393a-456b-9041-3389e96656fc-kube-api-access-rpfbk\") pod \"ingress-canary-fk7ll\" (UID: \"68ff79ec-393a-456b-9041-3389e96656fc\") " pod="openshift-ingress-canary/ingress-canary-fk7ll"
Aug 04 18:00:31.104390 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.104355    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.104390 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.104389    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-image-registry-private-configuration\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.104578 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.104404    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-ca-trust-extracted\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.104578 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.104420    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-certificates\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.104578 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.104445    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-bound-sa-token\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.104578 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:31.104501    2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:31.104578 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:31.104512    2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54954fb884-m76mm: secret "image-registry-tls" not found
Aug 04 18:00:31.104578 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:31.104556    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls podName:227c9ed0-d258-49e8-bbc5-01e00f6b1ede nodeName:}" failed. No retries permitted until 2026-08-04 18:00:31.604540367 +0000 UTC m=+34.575184699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls") pod "image-registry-54954fb884-m76mm" (UID: "227c9ed0-d258-49e8-bbc5-01e00f6b1ede") : secret "image-registry-tls" not found
Aug 04 18:00:31.104578 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.104574    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62c75b76-1fdb-4555-a35e-78d0d361c064-tmp-dir\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:31.104867 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.104599    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbrtb\" (UniqueName: \"kubernetes.io/projected/62c75b76-1fdb-4555-a35e-78d0d361c064-kube-api-access-tbrtb\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:31.104867 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.104636    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62c75b76-1fdb-4555-a35e-78d0d361c064-config-volume\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:31.104867 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.104657    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:31.104867 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.104687    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-trusted-ca\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.104867 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.104708    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtzdg\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-kube-api-access-dtzdg\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.104867 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.104784    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-ca-trust-extracted\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.104867 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:31.104867    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:31.105152 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:31.104911    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls podName:62c75b76-1fdb-4555-a35e-78d0d361c064 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:31.604895411 +0000 UTC m=+34.575539749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls") pod "dns-default-r9ntf" (UID: "62c75b76-1fdb-4555-a35e-78d0d361c064") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:31.105152 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.105053    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-certificates\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.105239 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.105149    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62c75b76-1fdb-4555-a35e-78d0d361c064-tmp-dir\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:31.105357 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.105339    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62c75b76-1fdb-4555-a35e-78d0d361c064-config-volume\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:31.106255 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.106231    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-trusted-ca\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.108087 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.108059    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-image-registry-private-configuration\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.108087 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.108069    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68ff79ec-393a-456b-9041-3389e96656fc-cert\") pod \"ingress-canary-fk7ll\" (UID: \"68ff79ec-393a-456b-9041-3389e96656fc\") " pod="openshift-ingress-canary/ingress-canary-fk7ll"
Aug 04 18:00:31.108243 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.108089    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-installation-pull-secrets\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.118839 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.118816    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtzdg\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-kube-api-access-dtzdg\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.119501 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.119480    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-bound-sa-token\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.119926 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.119910    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpfbk\" (UniqueName: \"kubernetes.io/projected/68ff79ec-393a-456b-9041-3389e96656fc-kube-api-access-rpfbk\") pod \"ingress-canary-fk7ll\" (UID: \"68ff79ec-393a-456b-9041-3389e96656fc\") " pod="openshift-ingress-canary/ingress-canary-fk7ll"
Aug 04 18:00:31.120618 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.120601    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbrtb\" (UniqueName: \"kubernetes.io/projected/62c75b76-1fdb-4555-a35e-78d0d361c064-kube-api-access-tbrtb\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:31.174417 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.174385    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:31.188298 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.188278    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fk7ll"
Aug 04 18:00:31.274401 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.274369    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:31.333823 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.333797    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fk7ll"]
Aug 04 18:00:31.338841 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:00:31.338802    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ff79ec_393a_456b_9041_3389e96656fc.slice/crio-c72b2739c32a18ff9fd46049e4c7d9ef0213d2fce0e7c0d4ea2a0079d42060dc WatchSource:0}: Error finding container c72b2739c32a18ff9fd46049e4c7d9ef0213d2fce0e7c0d4ea2a0079d42060dc: Status 404 returned error can't find the container with id c72b2739c32a18ff9fd46049e4c7d9ef0213d2fce0e7c0d4ea2a0079d42060dc
Aug 04 18:00:31.569335 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.569261    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:00:31.569335 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.569303    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:00:31.569491 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.569261    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:00:31.572586 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.572564    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:00:31.572700 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.572676    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:00:31.572809 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.572789    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 18:00:31.572896 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.572881    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 18:00:31.572945 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.572904    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nfg9c\""
Aug 04 18:00:31.573551 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.573532    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hp4qc\""
Aug 04 18:00:31.609600 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.609577    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:31.609673 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.609632    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:31.609732 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:31.609710    2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:31.609732 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:31.609723    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:31.609732 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:31.609727    2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54954fb884-m76mm: secret "image-registry-tls" not found
Aug 04 18:00:31.609843 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:31.609773    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls podName:62c75b76-1fdb-4555-a35e-78d0d361c064 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:32.609757721 +0000 UTC m=+35.580402057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls") pod "dns-default-r9ntf" (UID: "62c75b76-1fdb-4555-a35e-78d0d361c064") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:31.609843 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:31.609787    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls podName:227c9ed0-d258-49e8-bbc5-01e00f6b1ede nodeName:}" failed. No retries permitted until 2026-08-04 18:00:32.609781218 +0000 UTC m=+35.580425537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls") pod "image-registry-54954fb884-m76mm" (UID: "227c9ed0-d258-49e8-bbc5-01e00f6b1ede") : secret "image-registry-tls" not found
Aug 04 18:00:31.796045 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:31.796018    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fk7ll" event={"ID":"68ff79ec-393a-456b-9041-3389e96656fc","Type":"ContainerStarted","Data":"c72b2739c32a18ff9fd46049e4c7d9ef0213d2fce0e7c0d4ea2a0079d42060dc"}
Aug 04 18:00:32.617025 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:32.616989    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:32.617564 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:32.617068    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:32.617564 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:32.617140    2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:32.617564 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:32.617161    2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54954fb884-m76mm: secret "image-registry-tls" not found
Aug 04 18:00:32.617564 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:32.617204    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:32.617564 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:32.617221    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls podName:227c9ed0-d258-49e8-bbc5-01e00f6b1ede nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.617203033 +0000 UTC m=+37.587847355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls") pod "image-registry-54954fb884-m76mm" (UID: "227c9ed0-d258-49e8-bbc5-01e00f6b1ede") : secret "image-registry-tls" not found
Aug 04 18:00:32.617564 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:32.617253    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls podName:62c75b76-1fdb-4555-a35e-78d0d361c064 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.617238529 +0000 UTC m=+37.587882849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls") pod "dns-default-r9ntf" (UID: "62c75b76-1fdb-4555-a35e-78d0d361c064") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:34.632381 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:34.632135    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:34.632834 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:34.632288    2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:34.632834 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:34.632476    2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54954fb884-m76mm: secret "image-registry-tls" not found
Aug 04 18:00:34.632834 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:34.632450    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:34.632834 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:34.632534    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:34.632834 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:34.632546    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls podName:227c9ed0-d258-49e8-bbc5-01e00f6b1ede nodeName:}" failed. No retries permitted until 2026-08-04 18:00:38.632525069 +0000 UTC m=+41.603169418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls") pod "image-registry-54954fb884-m76mm" (UID: "227c9ed0-d258-49e8-bbc5-01e00f6b1ede") : secret "image-registry-tls" not found
Aug 04 18:00:34.632834 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:34.632587    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls podName:62c75b76-1fdb-4555-a35e-78d0d361c064 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:38.632571924 +0000 UTC m=+41.603216243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls") pod "dns-default-r9ntf" (UID: "62c75b76-1fdb-4555-a35e-78d0d361c064") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:35.427994 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:35.427965    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:36.807966 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:36.807936    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fk7ll" event={"ID":"68ff79ec-393a-456b-9041-3389e96656fc","Type":"ContainerStarted","Data":"7256a69aadbe1c79f21ff8df8e18850bfd9c6a4f3d8a93e1706bffc85dfd9ff1"}
Aug 04 18:00:36.813856 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:36.813822    2543 generic.go:356] "Generic (PLEG): container finished" podID="d3889c85-d79f-4629-94bb-c3fe1c1471ff" containerID="3c08c05502da7ef06ba261064d078a1c398825644956e3e23eb3bf631e5ac5d7" exitCode=0
Aug 04 18:00:36.813995 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:36.813872    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zthc" event={"ID":"d3889c85-d79f-4629-94bb-c3fe1c1471ff","Type":"ContainerDied","Data":"3c08c05502da7ef06ba261064d078a1c398825644956e3e23eb3bf631e5ac5d7"}
Aug 04 18:00:36.830352 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:36.830287    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fk7ll" podStartSLOduration=2.383177373 podStartE2EDuration="6.830273537s" totalImagesPullingTime="4.447096164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:36.829964229 +0000 UTC m=+39.800608570" watchObservedRunningTime="2026-08-04 18:00:36.830273537 +0000 UTC m=+39.800917876"
Aug 04 18:00:37.089499 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:37.089446    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:37.149354 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:37.149330    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:37.818689 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:37.818666    2543 generic.go:356] "Generic (PLEG): container finished" podID="d3889c85-d79f-4629-94bb-c3fe1c1471ff" containerID="be2f3c37c19606df8827c5c6494935cbe81cd2bf765cdd45001b25e51bbf81fe" exitCode=0
Aug 04 18:00:37.818973 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:37.818754    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zthc" event={"ID":"d3889c85-d79f-4629-94bb-c3fe1c1471ff","Type":"ContainerDied","Data":"be2f3c37c19606df8827c5c6494935cbe81cd2bf765cdd45001b25e51bbf81fe"}
Aug 04 18:00:38.661358 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:38.661335    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:38.661463 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:38.661384    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:38.661463 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:38.661451    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:38.661530 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:38.661466    2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:38.661530 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:38.661481    2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54954fb884-m76mm: secret "image-registry-tls" not found
Aug 04 18:00:38.661530 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:38.661489    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls podName:62c75b76-1fdb-4555-a35e-78d0d361c064 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:46.661477136 +0000 UTC m=+49.632121455 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls") pod "dns-default-r9ntf" (UID: "62c75b76-1fdb-4555-a35e-78d0d361c064") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:38.661530 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:38.661516    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls podName:227c9ed0-d258-49e8-bbc5-01e00f6b1ede nodeName:}" failed. No retries permitted until 2026-08-04 18:00:46.661503198 +0000 UTC m=+49.632147517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls") pod "image-registry-54954fb884-m76mm" (UID: "227c9ed0-d258-49e8-bbc5-01e00f6b1ede") : secret "image-registry-tls" not found
Aug 04 18:00:38.823760 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:38.823736    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zthc" event={"ID":"d3889c85-d79f-4629-94bb-c3fe1c1471ff","Type":"ContainerStarted","Data":"5d8f12d2bc825c64ea356610d95e405548abb699409e0f034ce281e734632989"}
Aug 04 18:00:38.855243 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:38.855194    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2zthc" podStartSLOduration=8.16315772 podStartE2EDuration="40.855175423s" totalImagesPullingTime="32.692017703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:38.853470091 +0000 UTC m=+41.824114429" watchObservedRunningTime="2026-08-04 18:00:38.855175423 +0000 UTC m=+41.825819758"
Aug 04 18:00:39.005764 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:39.005718    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:39.085768 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:39.085749    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:39.710847 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:39.710824    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:39.813451 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:39.813430    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:43.501300 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:43.501275    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:43.562991 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:43.562970    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:43.908812 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:43.908792    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:43.971457 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:43.971436    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:45.376419 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:45.376393    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:45.477856 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:45.477833    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:45.533712 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:45.533691    2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:46.714089 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:46.714067    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:00:46.714395 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:46.714112    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:00:46.714395 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:46.714203    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:46.714395 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:46.714210    2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:46.714395 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:46.714226    2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54954fb884-m76mm: secret "image-registry-tls" not found
Aug 04 18:00:46.714395 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:46.714254    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls podName:62c75b76-1fdb-4555-a35e-78d0d361c064 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:02.714242735 +0000 UTC m=+65.684887053 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls") pod "dns-default-r9ntf" (UID: "62c75b76-1fdb-4555-a35e-78d0d361c064") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:46.714395 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:00:46.714279    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls podName:227c9ed0-d258-49e8-bbc5-01e00f6b1ede nodeName:}" failed. No retries permitted until 2026-08-04 18:01:02.71426652 +0000 UTC m=+65.684910838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls") pod "image-registry-54954fb884-m76mm" (UID: "227c9ed0-d258-49e8-bbc5-01e00f6b1ede") : secret "image-registry-tls" not found
Aug 04 18:00:57.799591 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:00:57.799565    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkvl9"
Aug 04 18:01:02.316044 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.316012    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:01:02.316400 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.316067    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs\") pod \"network-metrics-daemon-r27tq\" (UID: \"4fea01c7-a50c-40f7-a628-9745002f5158\") " pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:01:02.319072 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.319054    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:01:02.319182 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.319154    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:01:02.327174 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:02.327156    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Aug 04 18:01:02.327255 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:02.327204    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs podName:4fea01c7-a50c-40f7-a628-9745002f5158 nodeName:}" failed. No retries permitted until 2026-08-04 18:02:06.327190266 +0000 UTC m=+129.297834584 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs") pod "network-metrics-daemon-r27tq" (UID: "4fea01c7-a50c-40f7-a628-9745002f5158") : secret "metrics-daemon-secret" not found
Aug 04 18:01:02.329726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.329708    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59896da8-3715-4ab5-b4b7-cbce0026cc98-original-pull-secret\") pod \"global-pull-secret-syncer-gsbtv\" (UID: \"59896da8-3715-4ab5-b4b7-cbce0026cc98\") " pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:01:02.416407 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.416384    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb5nx\" (UniqueName: \"kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx\") pod \"network-check-target-jzdgf\" (UID: \"d69a269d-b7ef-4618-aafb-475fc3de7ab2\") " pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:01:02.419418 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.419402    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 18:01:02.429210 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.429193    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 18:01:02.439125 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.439105    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb5nx\" (UniqueName: \"kubernetes.io/projected/d69a269d-b7ef-4618-aafb-475fc3de7ab2-kube-api-access-lb5nx\") pod \"network-check-target-jzdgf\" (UID: \"d69a269d-b7ef-4618-aafb-475fc3de7ab2\") " pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:01:02.490512 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.490498    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gsbtv"
Aug 04 18:01:02.500171 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.500145    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nfg9c\""
Aug 04 18:01:02.507224 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.507209    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:01:02.612488 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.612461    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gsbtv"]
Aug 04 18:01:02.615347 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:01:02.615296    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59896da8_3715_4ab5_b4b7_cbce0026cc98.slice/crio-b4fab9f7f5145140d5052c12b4dedc4f686138c1850e962485f5ca6681fa237c WatchSource:0}: Error finding container b4fab9f7f5145140d5052c12b4dedc4f686138c1850e962485f5ca6681fa237c: Status 404 returned error can't find the container with id b4fab9f7f5145140d5052c12b4dedc4f686138c1850e962485f5ca6681fa237c
Aug 04 18:01:02.632084 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.632060    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jzdgf"]
Aug 04 18:01:02.634467 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:01:02.634444    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69a269d_b7ef_4618_aafb_475fc3de7ab2.slice/crio-1e1a0dafb7b1932e0c746b45c0d06a301405f79a4a080c9ccfdedbb949d9513f WatchSource:0}: Error finding container 1e1a0dafb7b1932e0c746b45c0d06a301405f79a4a080c9ccfdedbb949d9513f: Status 404 returned error can't find the container with id 1e1a0dafb7b1932e0c746b45c0d06a301405f79a4a080c9ccfdedbb949d9513f
Aug 04 18:01:02.718324 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.718287    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:01:02.718411 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.718348    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:01:02.718446 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:02.718417    2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:01:02.718446 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:02.718433    2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54954fb884-m76mm: secret "image-registry-tls" not found
Aug 04 18:01:02.718446 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:02.718437    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:01:02.718530 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:02.718473    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls podName:227c9ed0-d258-49e8-bbc5-01e00f6b1ede nodeName:}" failed. No retries permitted until 2026-08-04 18:01:34.718460049 +0000 UTC m=+97.689104367 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls") pod "image-registry-54954fb884-m76mm" (UID: "227c9ed0-d258-49e8-bbc5-01e00f6b1ede") : secret "image-registry-tls" not found
Aug 04 18:01:02.718530 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:02.718485    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls podName:62c75b76-1fdb-4555-a35e-78d0d361c064 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:34.718479234 +0000 UTC m=+97.689123553 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls") pod "dns-default-r9ntf" (UID: "62c75b76-1fdb-4555-a35e-78d0d361c064") : secret "dns-default-metrics-tls" not found
Aug 04 18:01:02.871114 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.871089    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gsbtv" event={"ID":"59896da8-3715-4ab5-b4b7-cbce0026cc98","Type":"ContainerStarted","Data":"b4fab9f7f5145140d5052c12b4dedc4f686138c1850e962485f5ca6681fa237c"}
Aug 04 18:01:02.872111 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:02.872092    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jzdgf" event={"ID":"d69a269d-b7ef-4618-aafb-475fc3de7ab2","Type":"ContainerStarted","Data":"1e1a0dafb7b1932e0c746b45c0d06a301405f79a4a080c9ccfdedbb949d9513f"}
Aug 04 18:01:05.881207 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:05.881166    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jzdgf" event={"ID":"d69a269d-b7ef-4618-aafb-475fc3de7ab2","Type":"ContainerStarted","Data":"c29a3268c65d0ea4caf22162de8848700cdc746449e537613018e0ed94ff5e62"}
Aug 04 18:01:05.881673 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:05.881371    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:01:05.902063 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:05.902012    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jzdgf" podStartSLOduration=65.390505443 podStartE2EDuration="1m7.901995274s" totalImagesPullingTime="2.511489831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:05.901628386 +0000 UTC m=+68.872272739" watchObservedRunningTime="2026-08-04 18:01:05.901995274 +0000 UTC m=+68.872639607"
Aug 04 18:01:08.889965 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:08.889931    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gsbtv" event={"ID":"59896da8-3715-4ab5-b4b7-cbce0026cc98","Type":"ContainerStarted","Data":"7ad7ee672d7754e9148fbaf800ec15e37806d54cf1cd864872fb7b42a9c966fd"}
Aug 04 18:01:08.915947 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:08.915904    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gsbtv" podStartSLOduration=65.369024646 podStartE2EDuration="1m10.915891858s" totalImagesPullingTime="5.546867212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:08.91524305 +0000 UTC m=+71.885887394" watchObservedRunningTime="2026-08-04 18:01:08.915891858 +0000 UTC m=+71.886536196"
Aug 04 18:01:10.010947 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.010924    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-2txjf"]
Aug 04 18:01:10.031209 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.031179    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-2txjf"]
Aug 04 18:01:10.031331 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.031249    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2txjf"
Aug 04 18:01:10.034393 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.034375    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rhpn9\""
Aug 04 18:01:10.069682 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.069657    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glk7n\" (UniqueName: \"kubernetes.io/projected/966b1943-a685-4cd3-8f86-3bb210670312-kube-api-access-glk7n\") pod \"network-check-source-65bdbbff45-2txjf\" (UID: \"966b1943-a685-4cd3-8f86-3bb210670312\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2txjf"
Aug 04 18:01:10.100152 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.100122    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-cvxwr"]
Aug 04 18:01:10.123807 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.123784    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m"]
Aug 04 18:01:10.123912 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.123895    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.127927 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.127911    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Aug 04 18:01:10.128611 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.128594    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Aug 04 18:01:10.128877 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.128862    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-kk5td\""
Aug 04 18:01:10.129150 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.129126    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 04 18:01:10.129272 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.129251    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 04 18:01:10.139076 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.137862    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-wftvq"]
Aug 04 18:01:10.139183 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.139164    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m"
Aug 04 18:01:10.139923 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.139906    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Aug 04 18:01:10.141807 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.141787    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Aug 04 18:01:10.141807 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.141800    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:01:10.141942 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.141852    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rfkvs\""
Aug 04 18:01:10.141942 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.141901    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Aug 04 18:01:10.141942 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.141919    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Aug 04 18:01:10.160060 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.160045    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-cvxwr"]
Aug 04 18:01:10.160148 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.160068    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-wftvq"]
Aug 04 18:01:10.160148 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.160080    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m"]
Aug 04 18:01:10.160148 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.160143    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:10.162867 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.162851    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:01:10.163851 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.163833    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Aug 04 18:01:10.163927 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.163913    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xpbdv\""
Aug 04 18:01:10.164790 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.164770    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Aug 04 18:01:10.164894 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.164801    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Aug 04 18:01:10.167796 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.167775    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Aug 04 18:01:10.170541 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.170523    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glk7n\" (UniqueName: \"kubernetes.io/projected/966b1943-a685-4cd3-8f86-3bb210670312-kube-api-access-glk7n\") pod \"network-check-source-65bdbbff45-2txjf\" (UID: \"966b1943-a685-4cd3-8f86-3bb210670312\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2txjf"
Aug 04 18:01:10.170628 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.170553    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-tmp\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.170690 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.170645    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e3c4757-7362-4733-a723-36271decfb8c-config\") pod \"service-ca-operator-784f97bcf6-kgw2m\" (UID: \"9e3c4757-7362-4733-a723-36271decfb8c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m"
Aug 04 18:01:10.170744 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.170672    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-snapshots\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.170744 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.170737    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-serving-cert\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.170840 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.170765    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dxst\" (UniqueName: \"kubernetes.io/projected/57c7e8ed-7892-40d4-aebe-ccd866618ccc-kube-api-access-7dxst\") pod \"console-operator-5ccd6b445f-wftvq\" (UID: \"57c7e8ed-7892-40d4-aebe-ccd866618ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:10.170840 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.170787    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57c7e8ed-7892-40d4-aebe-ccd866618ccc-serving-cert\") pod \"console-operator-5ccd6b445f-wftvq\" (UID: \"57c7e8ed-7892-40d4-aebe-ccd866618ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:10.170840 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.170836    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57c7e8ed-7892-40d4-aebe-ccd866618ccc-trusted-ca\") pod \"console-operator-5ccd6b445f-wftvq\" (UID: \"57c7e8ed-7892-40d4-aebe-ccd866618ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:10.170979 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.170880    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-service-ca-bundle\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.170979 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.170937    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57c7e8ed-7892-40d4-aebe-ccd866618ccc-config\") pod \"console-operator-5ccd6b445f-wftvq\" (UID: \"57c7e8ed-7892-40d4-aebe-ccd866618ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:10.171060 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.170999    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-trusted-ca-bundle\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.171093 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.171065    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e3c4757-7362-4733-a723-36271decfb8c-serving-cert\") pod \"service-ca-operator-784f97bcf6-kgw2m\" (UID: \"9e3c4757-7362-4733-a723-36271decfb8c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m"
Aug 04 18:01:10.171093 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.171086    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgktc\" (UniqueName: \"kubernetes.io/projected/9e3c4757-7362-4733-a723-36271decfb8c-kube-api-access-jgktc\") pod \"service-ca-operator-784f97bcf6-kgw2m\" (UID: \"9e3c4757-7362-4733-a723-36271decfb8c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m"
Aug 04 18:01:10.171159 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.171104    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhchz\" (UniqueName: \"kubernetes.io/projected/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-kube-api-access-xhchz\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.179977 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.179958    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glk7n\" (UniqueName: \"kubernetes.io/projected/966b1943-a685-4cd3-8f86-3bb210670312-kube-api-access-glk7n\") pod \"network-check-source-65bdbbff45-2txjf\" (UID: \"966b1943-a685-4cd3-8f86-3bb210670312\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2txjf"
Aug 04 18:01:10.271702 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.271655    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-tmp\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.271702 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.271680    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e3c4757-7362-4733-a723-36271decfb8c-config\") pod \"service-ca-operator-784f97bcf6-kgw2m\" (UID: \"9e3c4757-7362-4733-a723-36271decfb8c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m"
Aug 04 18:01:10.271702 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.271697    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-snapshots\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.271869 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.271721    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-serving-cert\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.271869 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.271753    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dxst\" (UniqueName: \"kubernetes.io/projected/57c7e8ed-7892-40d4-aebe-ccd866618ccc-kube-api-access-7dxst\") pod \"console-operator-5ccd6b445f-wftvq\" (UID: \"57c7e8ed-7892-40d4-aebe-ccd866618ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:10.271869 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.271785    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57c7e8ed-7892-40d4-aebe-ccd866618ccc-serving-cert\") pod \"console-operator-5ccd6b445f-wftvq\" (UID: \"57c7e8ed-7892-40d4-aebe-ccd866618ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:10.271869 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.271807    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57c7e8ed-7892-40d4-aebe-ccd866618ccc-trusted-ca\") pod \"console-operator-5ccd6b445f-wftvq\" (UID: \"57c7e8ed-7892-40d4-aebe-ccd866618ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:10.271869 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.271834    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-service-ca-bundle\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.271869 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.271869    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57c7e8ed-7892-40d4-aebe-ccd866618ccc-config\") pod \"console-operator-5ccd6b445f-wftvq\" (UID: \"57c7e8ed-7892-40d4-aebe-ccd866618ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:10.272154 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.271892    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-trusted-ca-bundle\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.272154 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.271929    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e3c4757-7362-4733-a723-36271decfb8c-serving-cert\") pod \"service-ca-operator-784f97bcf6-kgw2m\" (UID: \"9e3c4757-7362-4733-a723-36271decfb8c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m"
Aug 04 18:01:10.272154 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.271956    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgktc\" (UniqueName: \"kubernetes.io/projected/9e3c4757-7362-4733-a723-36271decfb8c-kube-api-access-jgktc\") pod \"service-ca-operator-784f97bcf6-kgw2m\" (UID: \"9e3c4757-7362-4733-a723-36271decfb8c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m"
Aug 04 18:01:10.272154 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.271979    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhchz\" (UniqueName: \"kubernetes.io/projected/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-kube-api-access-xhchz\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.272154 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.272065    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-tmp\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.272429 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.272345    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e3c4757-7362-4733-a723-36271decfb8c-config\") pod \"service-ca-operator-784f97bcf6-kgw2m\" (UID: \"9e3c4757-7362-4733-a723-36271decfb8c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m"
Aug 04 18:01:10.272429 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.272398    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-snapshots\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.272864 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.272659    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-service-ca-bundle\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.272864 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.272702    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57c7e8ed-7892-40d4-aebe-ccd866618ccc-trusted-ca\") pod \"console-operator-5ccd6b445f-wftvq\" (UID: \"57c7e8ed-7892-40d4-aebe-ccd866618ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:10.272993 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.272900    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57c7e8ed-7892-40d4-aebe-ccd866618ccc-config\") pod \"console-operator-5ccd6b445f-wftvq\" (UID: \"57c7e8ed-7892-40d4-aebe-ccd866618ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:10.272993 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.272933    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-trusted-ca-bundle\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.274048 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.274026    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-serving-cert\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.274133 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.274100    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e3c4757-7362-4733-a723-36271decfb8c-serving-cert\") pod \"service-ca-operator-784f97bcf6-kgw2m\" (UID: \"9e3c4757-7362-4733-a723-36271decfb8c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m"
Aug 04 18:01:10.274526 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.274511    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57c7e8ed-7892-40d4-aebe-ccd866618ccc-serving-cert\") pod \"console-operator-5ccd6b445f-wftvq\" (UID: \"57c7e8ed-7892-40d4-aebe-ccd866618ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:10.282151 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.282132    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhchz\" (UniqueName: \"kubernetes.io/projected/7b6607f0-aea9-4a84-a327-6794dd2b7dd5-kube-api-access-xhchz\") pod \"insights-operator-77696685b-cvxwr\" (UID: \"7b6607f0-aea9-4a84-a327-6794dd2b7dd5\") " pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.282380 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.282362    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dxst\" (UniqueName: \"kubernetes.io/projected/57c7e8ed-7892-40d4-aebe-ccd866618ccc-kube-api-access-7dxst\") pod \"console-operator-5ccd6b445f-wftvq\" (UID: \"57c7e8ed-7892-40d4-aebe-ccd866618ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:10.283388 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.283369    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgktc\" (UniqueName: \"kubernetes.io/projected/9e3c4757-7362-4733-a723-36271decfb8c-kube-api-access-jgktc\") pod \"service-ca-operator-784f97bcf6-kgw2m\" (UID: \"9e3c4757-7362-4733-a723-36271decfb8c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m"
Aug 04 18:01:10.339283 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.339260    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2txjf"
Aug 04 18:01:10.432842 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.432820    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-cvxwr"
Aug 04 18:01:10.446589 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.446567    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m"
Aug 04 18:01:10.467972 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.467944    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:10.481921 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.481898    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-2txjf"]
Aug 04 18:01:10.485931 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:01:10.485900    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966b1943_a685_4cd3_8f86_3bb210670312.slice/crio-2ba24d6bb6545f067f0f460471c8e9d61079584f1b67110200b329c19b74202e WatchSource:0}: Error finding container 2ba24d6bb6545f067f0f460471c8e9d61079584f1b67110200b329c19b74202e: Status 404 returned error can't find the container with id 2ba24d6bb6545f067f0f460471c8e9d61079584f1b67110200b329c19b74202e
Aug 04 18:01:10.588544 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.588513    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-cvxwr"]
Aug 04 18:01:10.592128 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:01:10.592101    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6607f0_aea9_4a84_a327_6794dd2b7dd5.slice/crio-8da35acc253724228002e0cc43ee8d8d65d7b598cedb1f0a61ab06fb62c5788a WatchSource:0}: Error finding container 8da35acc253724228002e0cc43ee8d8d65d7b598cedb1f0a61ab06fb62c5788a: Status 404 returned error can't find the container with id 8da35acc253724228002e0cc43ee8d8d65d7b598cedb1f0a61ab06fb62c5788a
Aug 04 18:01:10.610206 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.610184    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m"]
Aug 04 18:01:10.613322 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:01:10.613279    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3c4757_7362_4733_a723_36271decfb8c.slice/crio-9ab63bf3a4628baef222bcbdc1a63675b0c6f3bde3683ee08074c9e31375bfce WatchSource:0}: Error finding container 9ab63bf3a4628baef222bcbdc1a63675b0c6f3bde3683ee08074c9e31375bfce: Status 404 returned error can't find the container with id 9ab63bf3a4628baef222bcbdc1a63675b0c6f3bde3683ee08074c9e31375bfce
Aug 04 18:01:10.631773 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.631715    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-wftvq"]
Aug 04 18:01:10.635723 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:01:10.635702    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c7e8ed_7892_40d4_aebe_ccd866618ccc.slice/crio-fd02d69d6fb7d944a26115365661c127700d44b75d027b7a4ba8d6f55fcf1fe9 WatchSource:0}: Error finding container fd02d69d6fb7d944a26115365661c127700d44b75d027b7a4ba8d6f55fcf1fe9: Status 404 returned error can't find the container with id fd02d69d6fb7d944a26115365661c127700d44b75d027b7a4ba8d6f55fcf1fe9
Aug 04 18:01:10.894893 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.894865    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-cvxwr" event={"ID":"7b6607f0-aea9-4a84-a327-6794dd2b7dd5","Type":"ContainerStarted","Data":"8da35acc253724228002e0cc43ee8d8d65d7b598cedb1f0a61ab06fb62c5788a"}
Aug 04 18:01:10.895831 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.895811    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m" event={"ID":"9e3c4757-7362-4733-a723-36271decfb8c","Type":"ContainerStarted","Data":"9ab63bf3a4628baef222bcbdc1a63675b0c6f3bde3683ee08074c9e31375bfce"}
Aug 04 18:01:10.896951 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.896933    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq" event={"ID":"57c7e8ed-7892-40d4-aebe-ccd866618ccc","Type":"ContainerStarted","Data":"fd02d69d6fb7d944a26115365661c127700d44b75d027b7a4ba8d6f55fcf1fe9"}
Aug 04 18:01:10.898255 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.898234    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2txjf" event={"ID":"966b1943-a685-4cd3-8f86-3bb210670312","Type":"ContainerStarted","Data":"817ae59c712d6e5f25149e6a8cebab1e7780e2baaf15f78ecd5961c68bfc9b61"}
Aug 04 18:01:10.898343 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.898272    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2txjf" event={"ID":"966b1943-a685-4cd3-8f86-3bb210670312","Type":"ContainerStarted","Data":"2ba24d6bb6545f067f0f460471c8e9d61079584f1b67110200b329c19b74202e"}
Aug 04 18:01:10.921992 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:10.921956    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2txjf" podStartSLOduration=1.921944784 podStartE2EDuration="1.921944784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:10.92191745 +0000 UTC m=+73.892561790" watchObservedRunningTime="2026-08-04 18:01:10.921944784 +0000 UTC m=+73.892589122"
Aug 04 18:01:13.909091 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:13.909036    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/0.log"
Aug 04 18:01:13.909091 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:13.909080    2543 generic.go:356] "Generic (PLEG): container finished" podID="57c7e8ed-7892-40d4-aebe-ccd866618ccc" containerID="e313435e331c6f5af0ee7785cf56d93d407ff9fd9c78c2bd56298d48943b3876" exitCode=255
Aug 04 18:01:13.909492 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:13.909174    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq" event={"ID":"57c7e8ed-7892-40d4-aebe-ccd866618ccc","Type":"ContainerDied","Data":"e313435e331c6f5af0ee7785cf56d93d407ff9fd9c78c2bd56298d48943b3876"}
Aug 04 18:01:13.909492 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:13.909367    2543 scope.go:117] "RemoveContainer" containerID="e313435e331c6f5af0ee7785cf56d93d407ff9fd9c78c2bd56298d48943b3876"
Aug 04 18:01:13.910827 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:13.910794    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-cvxwr" event={"ID":"7b6607f0-aea9-4a84-a327-6794dd2b7dd5","Type":"ContainerStarted","Data":"0a0a458434b56aeceb924aa46ceabc993b4867e954f5b7f3de12d6dfdc6668b9"}
Aug 04 18:01:13.912328 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:13.912291    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m" event={"ID":"9e3c4757-7362-4733-a723-36271decfb8c","Type":"ContainerStarted","Data":"673d2d3b5a3c2663e5972d7cb9082a08e8c9e61f057bf44d41fd15258d55c1ac"}
Aug 04 18:01:13.954156 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:13.954115    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-cvxwr" podStartSLOduration=0.900409638 podStartE2EDuration="3.95410191s" totalImagesPullingTime="3.053692272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:13.953150751 +0000 UTC m=+76.923795093" watchObservedRunningTime="2026-08-04 18:01:13.95410191 +0000 UTC m=+76.924746260"
Aug 04 18:01:13.970528 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:13.970482    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m" podStartSLOduration=0.939661304 podStartE2EDuration="3.970465763s" totalImagesPullingTime="3.030804459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:13.969634352 +0000 UTC m=+76.940278693" watchObservedRunningTime="2026-08-04 18:01:13.970465763 +0000 UTC m=+76.941110104"
Aug 04 18:01:14.917460 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:14.917393    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/1.log"
Aug 04 18:01:14.917844 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:14.917744    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/0.log"
Aug 04 18:01:14.917844 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:14.917774    2543 generic.go:356] "Generic (PLEG): container finished" podID="57c7e8ed-7892-40d4-aebe-ccd866618ccc" containerID="4a3a24b1b32d74580807d0462834898c59dc64faaca33463e19fa36b09f2a45d" exitCode=255
Aug 04 18:01:14.917844 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:14.917803    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq" event={"ID":"57c7e8ed-7892-40d4-aebe-ccd866618ccc","Type":"ContainerDied","Data":"4a3a24b1b32d74580807d0462834898c59dc64faaca33463e19fa36b09f2a45d"}
Aug 04 18:01:14.917993 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:14.917861    2543 scope.go:117] "RemoveContainer" containerID="e313435e331c6f5af0ee7785cf56d93d407ff9fd9c78c2bd56298d48943b3876"
Aug 04 18:01:14.918525 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:14.918473    2543 scope.go:117] "RemoveContainer" containerID="4a3a24b1b32d74580807d0462834898c59dc64faaca33463e19fa36b09f2a45d"
Aug 04 18:01:14.918785 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:14.918746    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-wftvq_openshift-console-operator(57c7e8ed-7892-40d4-aebe-ccd866618ccc)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq" podUID="57c7e8ed-7892-40d4-aebe-ccd866618ccc"
Aug 04 18:01:15.081545 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.081517    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w9k9t"]
Aug 04 18:01:15.084235 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.084207    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.086878 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.086853    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 04 18:01:15.086960 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.086910    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 04 18:01:15.086960 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.086936    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vx672\""
Aug 04 18:01:15.097831 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.097808    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w9k9t"]
Aug 04 18:01:15.104648 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.104625    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c1974c9-ee39-4f0d-bcae-c749b4a26519-crio-socket\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.104738 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.104663    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c1974c9-ee39-4f0d-bcae-c749b4a26519-data-volume\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.104738 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.104720    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.105001 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.104789    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.105001 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.104847    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr8sc\" (UniqueName: \"kubernetes.io/projected/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-api-access-lr8sc\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.205341 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.205279    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr8sc\" (UniqueName: \"kubernetes.io/projected/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-api-access-lr8sc\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.205414 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.205340    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c1974c9-ee39-4f0d-bcae-c749b4a26519-crio-socket\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.205414 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.205363    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c1974c9-ee39-4f0d-bcae-c749b4a26519-data-volume\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.205414 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.205387    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.205414 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.205409    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.205581 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:15.205471    2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:01:15.205581 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.205499    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c1974c9-ee39-4f0d-bcae-c749b4a26519-crio-socket\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.205581 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:15.205532    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm podName:2c1974c9-ee39-4f0d-bcae-c749b4a26519 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:15.705514442 +0000 UTC m=+78.676158781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9k9t" (UID: "2c1974c9-ee39-4f0d-bcae-c749b4a26519") : configmap "kube-rbac-proxy" not found
Aug 04 18:01:15.205581 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:15.205569    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:15.205718 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:15.205607    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls podName:2c1974c9-ee39-4f0d-bcae-c749b4a26519 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:15.705595654 +0000 UTC m=+78.676239980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9k9t" (UID: "2c1974c9-ee39-4f0d-bcae-c749b4a26519") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:15.205718 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.205646    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c1974c9-ee39-4f0d-bcae-c749b4a26519-data-volume\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.216944 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.216923    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr8sc\" (UniqueName: \"kubernetes.io/projected/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-api-access-lr8sc\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.709419 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.709394    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.709519 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.709458    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:15.709519 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:15.709508    2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:01:15.709583 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:15.709529    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:15.709583 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:15.709550    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm podName:2c1974c9-ee39-4f0d-bcae-c749b4a26519 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:16.709538788 +0000 UTC m=+79.680183106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9k9t" (UID: "2c1974c9-ee39-4f0d-bcae-c749b4a26519") : configmap "kube-rbac-proxy" not found
Aug 04 18:01:15.709583 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:15.709581    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls podName:2c1974c9-ee39-4f0d-bcae-c749b4a26519 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:16.709570213 +0000 UTC m=+79.680214532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9k9t" (UID: "2c1974c9-ee39-4f0d-bcae-c749b4a26519") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:15.922038 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.922020    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/1.log"
Aug 04 18:01:15.922333 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:15.922283    2543 scope.go:117] "RemoveContainer" containerID="4a3a24b1b32d74580807d0462834898c59dc64faaca33463e19fa36b09f2a45d"
Aug 04 18:01:15.922436 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:15.922418    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-wftvq_openshift-console-operator(57c7e8ed-7892-40d4-aebe-ccd866618ccc)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq" podUID="57c7e8ed-7892-40d4-aebe-ccd866618ccc"
Aug 04 18:01:16.716121 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:16.716097    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:16.716240 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:16.716129    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:16.716240 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:16.716207    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:16.716240 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:16.716217    2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:01:16.716401 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:16.716252    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls podName:2c1974c9-ee39-4f0d-bcae-c749b4a26519 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:18.716242055 +0000 UTC m=+81.686886374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9k9t" (UID: "2c1974c9-ee39-4f0d-bcae-c749b4a26519") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:16.716401 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:16.716290    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm podName:2c1974c9-ee39-4f0d-bcae-c749b4a26519 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:18.716271425 +0000 UTC m=+81.686915746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9k9t" (UID: "2c1974c9-ee39-4f0d-bcae-c749b4a26519") : configmap "kube-rbac-proxy" not found
Aug 04 18:01:17.043260 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:17.043211    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p52sc_d05e7582-030c-4b50-b5ce-90e4c81c1f1e/dns-node-resolver/0.log"
Aug 04 18:01:18.042569 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:18.042550    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z8dv8_dea9d566-e866-4409-ad9a-108474118ce2/node-ca/0.log"
Aug 04 18:01:18.730478 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:18.730457    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:18.730782 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:18.730493    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:18.730782 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:18.730571    2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:01:18.730782 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:18.730605    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:18.730782 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:18.730630    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm podName:2c1974c9-ee39-4f0d-bcae-c749b4a26519 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:22.730616448 +0000 UTC m=+85.701260766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9k9t" (UID: "2c1974c9-ee39-4f0d-bcae-c749b4a26519") : configmap "kube-rbac-proxy" not found
Aug 04 18:01:18.730782 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:18.730654    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls podName:2c1974c9-ee39-4f0d-bcae-c749b4a26519 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:22.730643862 +0000 UTC m=+85.701288180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9k9t" (UID: "2c1974c9-ee39-4f0d-bcae-c749b4a26519") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:18.843715 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:18.843695    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fk7ll_68ff79ec-393a-456b-9041-3389e96656fc/serve-healthcheck-canary/0.log"
Aug 04 18:01:20.468215 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:20.468124    2543 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:20.468215 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:20.468167    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:20.468606 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:20.468521    2543 scope.go:117] "RemoveContainer" containerID="4a3a24b1b32d74580807d0462834898c59dc64faaca33463e19fa36b09f2a45d"
Aug 04 18:01:20.468657 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:20.468640    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-wftvq_openshift-console-operator(57c7e8ed-7892-40d4-aebe-ccd866618ccc)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq" podUID="57c7e8ed-7892-40d4-aebe-ccd866618ccc"
Aug 04 18:01:22.754550 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:22.754518    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:22.754997 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:22.754556    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:22.754997 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:22.754628    2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:01:22.754997 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:22.754666    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:22.754997 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:22.754686    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm podName:2c1974c9-ee39-4f0d-bcae-c749b4a26519 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:30.754669239 +0000 UTC m=+93.725313576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9k9t" (UID: "2c1974c9-ee39-4f0d-bcae-c749b4a26519") : configmap "kube-rbac-proxy" not found
Aug 04 18:01:22.754997 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:22.754708    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls podName:2c1974c9-ee39-4f0d-bcae-c749b4a26519 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:30.754698768 +0000 UTC m=+93.725343091 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9k9t" (UID: "2c1974c9-ee39-4f0d-bcae-c749b4a26519") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:30.809620 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:30.809582    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:30.809973 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:30.809637    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:30.810047 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:30.810028    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c1974c9-ee39-4f0d-bcae-c749b4a26519-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:30.811701 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:30.811672    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c1974c9-ee39-4f0d-bcae-c749b4a26519-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9k9t\" (UID: \"2c1974c9-ee39-4f0d-bcae-c749b4a26519\") " pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:30.992323 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:30.992290    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w9k9t"
Aug 04 18:01:31.111707 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:31.111683    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w9k9t"]
Aug 04 18:01:31.114726 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:01:31.114698    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1974c9_ee39_4f0d_bcae_c749b4a26519.slice/crio-48ecf0cf58b6e8f8ccfd6363363a8e5455a20201c308a52ff87780b640a7e099 WatchSource:0}: Error finding container 48ecf0cf58b6e8f8ccfd6363363a8e5455a20201c308a52ff87780b640a7e099: Status 404 returned error can't find the container with id 48ecf0cf58b6e8f8ccfd6363363a8e5455a20201c308a52ff87780b640a7e099
Aug 04 18:01:31.963681 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:31.963655    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9k9t" event={"ID":"2c1974c9-ee39-4f0d-bcae-c749b4a26519","Type":"ContainerStarted","Data":"68d49bac66a00b55f2d738e38994a686d036ad7830ce3284db3d62f54d61fc44"}
Aug 04 18:01:31.963937 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:31.963704    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9k9t" event={"ID":"2c1974c9-ee39-4f0d-bcae-c749b4a26519","Type":"ContainerStarted","Data":"9f352a150a5529aa027185ee0c2d0e2c3cecfa8a9393e678a5c15ccc346cc164"}
Aug 04 18:01:31.963937 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:31.963722    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9k9t" event={"ID":"2c1974c9-ee39-4f0d-bcae-c749b4a26519","Type":"ContainerStarted","Data":"48ecf0cf58b6e8f8ccfd6363363a8e5455a20201c308a52ff87780b640a7e099"}
Aug 04 18:01:34.569857 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.569835    2543 scope.go:117] "RemoveContainer" containerID="4a3a24b1b32d74580807d0462834898c59dc64faaca33463e19fa36b09f2a45d"
Aug 04 18:01:34.735527 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.735501    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:01:34.735664 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.735570    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:01:34.738111 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.738085    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls\") pod \"image-registry-54954fb884-m76mm\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") " pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:01:34.738407 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.738387    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62c75b76-1fdb-4555-a35e-78d0d361c064-metrics-tls\") pod \"dns-default-r9ntf\" (UID: \"62c75b76-1fdb-4555-a35e-78d0d361c064\") " pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:01:34.772814 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.772788    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5b4qm\""
Aug 04 18:01:34.780569 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.780533    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:01:34.795827 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.795755    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k2cqb\""
Aug 04 18:01:34.804222 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.804094    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:01:34.942788 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.942759    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54954fb884-m76mm"]
Aug 04 18:01:34.946207 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:01:34.946181    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227c9ed0_d258_49e8_bbc5_01e00f6b1ede.slice/crio-3b4412fcad34f4ec37b7b59aeafdcf58acef97e3b26f39cdeb8d268547d7cf2b WatchSource:0}: Error finding container 3b4412fcad34f4ec37b7b59aeafdcf58acef97e3b26f39cdeb8d268547d7cf2b: Status 404 returned error can't find the container with id 3b4412fcad34f4ec37b7b59aeafdcf58acef97e3b26f39cdeb8d268547d7cf2b
Aug 04 18:01:34.957762 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.957736    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r9ntf"]
Aug 04 18:01:34.961887 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:01:34.961858    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c75b76_1fdb_4555_a35e_78d0d361c064.slice/crio-e7cf43089c4d8f2f85b6e25a8c7ae59481eae72a0c240cf04635229b9051fc8f WatchSource:0}: Error finding container e7cf43089c4d8f2f85b6e25a8c7ae59481eae72a0c240cf04635229b9051fc8f: Status 404 returned error can't find the container with id e7cf43089c4d8f2f85b6e25a8c7ae59481eae72a0c240cf04635229b9051fc8f
Aug 04 18:01:34.974830 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.974800    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54954fb884-m76mm" event={"ID":"227c9ed0-d258-49e8-bbc5-01e00f6b1ede","Type":"ContainerStarted","Data":"3b4412fcad34f4ec37b7b59aeafdcf58acef97e3b26f39cdeb8d268547d7cf2b"}
Aug 04 18:01:34.976898 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.976879    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/1.log"
Aug 04 18:01:34.976995 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.976974    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq" event={"ID":"57c7e8ed-7892-40d4-aebe-ccd866618ccc","Type":"ContainerStarted","Data":"b511ab04cf864f0664aab594bbdef061d3bb65f11e4732d1ca2f6130e070e634"}
Aug 04 18:01:34.977272 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.977248    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:34.978545 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.978525    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9ntf" event={"ID":"62c75b76-1fdb-4555-a35e-78d0d361c064","Type":"ContainerStarted","Data":"e7cf43089c4d8f2f85b6e25a8c7ae59481eae72a0c240cf04635229b9051fc8f"}
Aug 04 18:01:34.996454 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:34.996411    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq" podStartSLOduration=21.990875377 podStartE2EDuration="24.996395259s" totalImagesPullingTime="3.005519882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:34.994937869 +0000 UTC m=+97.965582210" watchObservedRunningTime="2026-08-04 18:01:34.996395259 +0000 UTC m=+97.967039591"
Aug 04 18:01:35.355759 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:35.355730    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-wftvq"
Aug 04 18:01:35.983896 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:35.983781    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54954fb884-m76mm" event={"ID":"227c9ed0-d258-49e8-bbc5-01e00f6b1ede","Type":"ContainerStarted","Data":"a26bbeaf283b2e8e97d4d93c194e8fec5c40ad54dbede5373e27a5547f24277b"}
Aug 04 18:01:35.984324 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:35.984193    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:01:35.986007 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:35.985981    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9k9t" event={"ID":"2c1974c9-ee39-4f0d-bcae-c749b4a26519","Type":"ContainerStarted","Data":"d700e4d995bd142931f7d21b130aa2a749c5f481108cdb65e69fbda271b8bd85"}
Aug 04 18:01:36.011989 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:36.011945    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54954fb884-m76mm" podStartSLOduration=98.011929021 podStartE2EDuration="1m38.011929021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:36.011210013 +0000 UTC m=+98.981854354" watchObservedRunningTime="2026-08-04 18:01:36.011929021 +0000 UTC m=+98.982573360"
Aug 04 18:01:36.047801 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:36.047767    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w9k9t" podStartSLOduration=16.624054963 podStartE2EDuration="21.047753976s" totalImagesPullingTime="4.423699013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:01:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:36.045867459 +0000 UTC m=+99.016511834" watchObservedRunningTime="2026-08-04 18:01:36.047753976 +0000 UTC m=+99.018398317"
Aug 04 18:01:36.885913 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:36.885890    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jzdgf"
Aug 04 18:01:36.991115 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:36.991053    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9ntf" event={"ID":"62c75b76-1fdb-4555-a35e-78d0d361c064","Type":"ContainerStarted","Data":"a945b005e8d42b02617da3a33dff6db7e431771250bcf735b5f4317984528cb9"}
Aug 04 18:01:36.991115 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:36.991099    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9ntf" event={"ID":"62c75b76-1fdb-4555-a35e-78d0d361c064","Type":"ContainerStarted","Data":"647b2823a7b38b0b64f3aaddf99642aa049dba01e81bd31442662d933591b37f"}
Aug 04 18:01:36.991557 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:36.991355    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:01:37.012745 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:37.012706    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r9ntf" podStartSLOduration=65.269292313 podStartE2EDuration="1m7.012694909s" totalImagesPullingTime="1.743402596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:37.01194878 +0000 UTC m=+99.982593121" watchObservedRunningTime="2026-08-04 18:01:37.012694909 +0000 UTC m=+99.983339248"
Aug 04 18:01:39.129907 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.129874    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68586c4f6b-zqx88"]
Aug 04 18:01:39.144906 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.144876    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68586c4f6b-zqx88"
Aug 04 18:01:39.145044 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.144958    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68586c4f6b-zqx88"]
Aug 04 18:01:39.147675 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.147650    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cx6w5\""
Aug 04 18:01:39.147783 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.147651    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Aug 04 18:01:39.148948 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.148930    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Aug 04 18:01:39.149057 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.148968    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Aug 04 18:01:39.149057 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.149006    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Aug 04 18:01:39.168051 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.168026    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cgdc\" (UniqueName: \"kubernetes.io/projected/43ea57a5-870a-4bc7-8023-e7eeaafef37a-kube-api-access-8cgdc\") pod \"managed-serviceaccount-addon-agent-68586c4f6b-zqx88\" (UID: \"43ea57a5-870a-4bc7-8023-e7eeaafef37a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68586c4f6b-zqx88"
Aug 04 18:01:39.168152 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.168068    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43ea57a5-870a-4bc7-8023-e7eeaafef37a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-68586c4f6b-zqx88\" (UID: \"43ea57a5-870a-4bc7-8023-e7eeaafef37a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68586c4f6b-zqx88"
Aug 04 18:01:39.229297 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.229275    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-q6rn5"]
Aug 04 18:01:39.248750 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.248730    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-q6rn5"]
Aug 04 18:01:39.248869 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.248811    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-q6rn5"
Aug 04 18:01:39.251388 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.251369    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Aug 04 18:01:39.251481 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.251398    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Aug 04 18:01:39.251968 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.251941    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-d9sqf\""
Aug 04 18:01:39.268354 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.268335    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cgdc\" (UniqueName: \"kubernetes.io/projected/43ea57a5-870a-4bc7-8023-e7eeaafef37a-kube-api-access-8cgdc\") pod \"managed-serviceaccount-addon-agent-68586c4f6b-zqx88\" (UID: \"43ea57a5-870a-4bc7-8023-e7eeaafef37a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68586c4f6b-zqx88"
Aug 04 18:01:39.268442 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.268369    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43ea57a5-870a-4bc7-8023-e7eeaafef37a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-68586c4f6b-zqx88\" (UID: \"43ea57a5-870a-4bc7-8023-e7eeaafef37a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68586c4f6b-zqx88"
Aug 04 18:01:39.270832 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.270804    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43ea57a5-870a-4bc7-8023-e7eeaafef37a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-68586c4f6b-zqx88\" (UID: \"43ea57a5-870a-4bc7-8023-e7eeaafef37a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68586c4f6b-zqx88"
Aug 04 18:01:39.284748 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.284724    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cgdc\" (UniqueName: \"kubernetes.io/projected/43ea57a5-870a-4bc7-8023-e7eeaafef37a-kube-api-access-8cgdc\") pod \"managed-serviceaccount-addon-agent-68586c4f6b-zqx88\" (UID: \"43ea57a5-870a-4bc7-8023-e7eeaafef37a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68586c4f6b-zqx88"
Aug 04 18:01:39.368914 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.368893    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl69b\" (UniqueName: \"kubernetes.io/projected/3aa47b21-f214-40ef-a37d-29d68ba50805-kube-api-access-fl69b\") pod \"downloads-77c974ffcf-q6rn5\" (UID: \"3aa47b21-f214-40ef-a37d-29d68ba50805\") " pod="openshift-console/downloads-77c974ffcf-q6rn5"
Aug 04 18:01:39.461339 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.461280    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68586c4f6b-zqx88"
Aug 04 18:01:39.469526 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.469509    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl69b\" (UniqueName: \"kubernetes.io/projected/3aa47b21-f214-40ef-a37d-29d68ba50805-kube-api-access-fl69b\") pod \"downloads-77c974ffcf-q6rn5\" (UID: \"3aa47b21-f214-40ef-a37d-29d68ba50805\") " pod="openshift-console/downloads-77c974ffcf-q6rn5"
Aug 04 18:01:39.480149 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.480125    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl69b\" (UniqueName: \"kubernetes.io/projected/3aa47b21-f214-40ef-a37d-29d68ba50805-kube-api-access-fl69b\") pod \"downloads-77c974ffcf-q6rn5\" (UID: \"3aa47b21-f214-40ef-a37d-29d68ba50805\") " pod="openshift-console/downloads-77c974ffcf-q6rn5"
Aug 04 18:01:39.556831 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.556647    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-q6rn5"
Aug 04 18:01:39.608145 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.608125    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68586c4f6b-zqx88"]
Aug 04 18:01:39.610077 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:01:39.610055    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43ea57a5_870a_4bc7_8023_e7eeaafef37a.slice/crio-e4927e2e7ecc52edc91cc89f977fbfce15bbcb6e6f5f42b2ad3b9ddf29a9d387 WatchSource:0}: Error finding container e4927e2e7ecc52edc91cc89f977fbfce15bbcb6e6f5f42b2ad3b9ddf29a9d387: Status 404 returned error can't find the container with id e4927e2e7ecc52edc91cc89f977fbfce15bbcb6e6f5f42b2ad3b9ddf29a9d387
Aug 04 18:01:39.754774 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:01:39.754737    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa47b21_f214_40ef_a37d_29d68ba50805.slice/crio-9aadda23b5e9d4bd291398d99eff75efaa0ac89bb2d1c3978be6bf15285a4116 WatchSource:0}: Error finding container 9aadda23b5e9d4bd291398d99eff75efaa0ac89bb2d1c3978be6bf15285a4116: Status 404 returned error can't find the container with id 9aadda23b5e9d4bd291398d99eff75efaa0ac89bb2d1c3978be6bf15285a4116
Aug 04 18:01:39.757429 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:39.757408    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-q6rn5"]
Aug 04 18:01:40.001500 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:40.001474    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-q6rn5" event={"ID":"3aa47b21-f214-40ef-a37d-29d68ba50805","Type":"ContainerStarted","Data":"9aadda23b5e9d4bd291398d99eff75efaa0ac89bb2d1c3978be6bf15285a4116"}
Aug 04 18:01:40.002493 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:40.002452    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68586c4f6b-zqx88" event={"ID":"43ea57a5-870a-4bc7-8023-e7eeaafef37a","Type":"ContainerStarted","Data":"e4927e2e7ecc52edc91cc89f977fbfce15bbcb6e6f5f42b2ad3b9ddf29a9d387"}
Aug 04 18:01:43.016707 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:43.016672    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68586c4f6b-zqx88" event={"ID":"43ea57a5-870a-4bc7-8023-e7eeaafef37a","Type":"ContainerStarted","Data":"da3123284cc412e90cb8b0db6c87b5430b2d4315cb00d3c9682eb33773adfc1d"}
Aug 04 18:01:43.035766 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:43.035722    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68586c4f6b-zqx88" podStartSLOduration=1.44090415 podStartE2EDuration="4.035708543s" totalImagesPullingTime="2.594804393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:43.035169663 +0000 UTC m=+106.005814003" watchObservedRunningTime="2026-08-04 18:01:43.035708543 +0000 UTC m=+106.006352901"
Aug 04 18:01:44.292772 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.292735    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6944cbf757-l4rr2"]
Aug 04 18:01:44.294978 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.294952    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.297799 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.297768    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Aug 04 18:01:44.298876 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.298854    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Aug 04 18:01:44.299003 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.298969    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Aug 04 18:01:44.299096 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.299076    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Aug 04 18:01:44.299225 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.299204    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Aug 04 18:01:44.299298 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.299285    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-68557\""
Aug 04 18:01:44.314352 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.314327    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6944cbf757-l4rr2"]
Aug 04 18:01:44.408387 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.408357    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2m2f\" (UniqueName: \"kubernetes.io/projected/da619ae0-4438-48c9-b256-ef8d6c677b44-kube-api-access-d2m2f\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.408387 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.408395    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-service-ca\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.408627 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.408458    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da619ae0-4438-48c9-b256-ef8d6c677b44-console-oauth-config\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.408627 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.408523    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da619ae0-4438-48c9-b256-ef8d6c677b44-console-serving-cert\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.408627 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.408546    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-console-config\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.408627 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.408575    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-oauth-serving-cert\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.509364 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.509087    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da619ae0-4438-48c9-b256-ef8d6c677b44-console-serving-cert\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.509364 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.509131    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-console-config\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.509364 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.509168    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-oauth-serving-cert\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.509364 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.509218    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2m2f\" (UniqueName: \"kubernetes.io/projected/da619ae0-4438-48c9-b256-ef8d6c677b44-kube-api-access-d2m2f\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.509364 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.509248    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-service-ca\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.509364 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.509291    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da619ae0-4438-48c9-b256-ef8d6c677b44-console-oauth-config\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.510587 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.510559    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-oauth-serving-cert\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.511164 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.511141    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-console-config\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.511258 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.511221    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-service-ca\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.512361 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.512335    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da619ae0-4438-48c9-b256-ef8d6c677b44-console-oauth-config\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.513525 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.513502    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da619ae0-4438-48c9-b256-ef8d6c677b44-console-serving-cert\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.521685 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.521659    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2m2f\" (UniqueName: \"kubernetes.io/projected/da619ae0-4438-48c9-b256-ef8d6c677b44-kube-api-access-d2m2f\") pod \"console-6944cbf757-l4rr2\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") " pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.605111 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.605076    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:44.746412 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:44.746380    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6944cbf757-l4rr2"]
Aug 04 18:01:44.749561 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:01:44.749532    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda619ae0_4438_48c9_b256_ef8d6c677b44.slice/crio-6065522e3c05a1133659dcd72636c885b6eb4577a1b2b51cd497159f4c21e205 WatchSource:0}: Error finding container 6065522e3c05a1133659dcd72636c885b6eb4577a1b2b51cd497159f4c21e205: Status 404 returned error can't find the container with id 6065522e3c05a1133659dcd72636c885b6eb4577a1b2b51cd497159f4c21e205
Aug 04 18:01:45.025490 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:45.025416    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6944cbf757-l4rr2" event={"ID":"da619ae0-4438-48c9-b256-ef8d6c677b44","Type":"ContainerStarted","Data":"6065522e3c05a1133659dcd72636c885b6eb4577a1b2b51cd497159f4c21e205"}
Aug 04 18:01:45.047015 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:45.046983    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-k9ktl"]
Aug 04 18:01:45.049968 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:45.049949    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-k9ktl"
Aug 04 18:01:45.052524 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:45.052500    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Aug 04 18:01:45.052618 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:45.052581    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vfmkw\""
Aug 04 18:01:45.058413 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:45.058382    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-k9ktl"]
Aug 04 18:01:45.113970 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:45.113939    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/726c7c38-d14e-4db0-b34d-d8ad5bb965b1-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-k9ktl\" (UID: \"726c7c38-d14e-4db0-b34d-d8ad5bb965b1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-k9ktl"
Aug 04 18:01:45.214382 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:45.214341    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/726c7c38-d14e-4db0-b34d-d8ad5bb965b1-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-k9ktl\" (UID: \"726c7c38-d14e-4db0-b34d-d8ad5bb965b1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-k9ktl"
Aug 04 18:01:45.214546 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:45.214495    2543 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Aug 04 18:01:45.214592 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:01:45.214571    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/726c7c38-d14e-4db0-b34d-d8ad5bb965b1-tls-certificates podName:726c7c38-d14e-4db0-b34d-d8ad5bb965b1 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:45.71455551 +0000 UTC m=+108.685199828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/726c7c38-d14e-4db0-b34d-d8ad5bb965b1-tls-certificates") pod "prometheus-operator-admission-webhook-66685ff6d4-k9ktl" (UID: "726c7c38-d14e-4db0-b34d-d8ad5bb965b1") : secret "prometheus-operator-admission-webhook-tls" not found
Aug 04 18:01:45.719665 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:45.719602    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/726c7c38-d14e-4db0-b34d-d8ad5bb965b1-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-k9ktl\" (UID: \"726c7c38-d14e-4db0-b34d-d8ad5bb965b1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-k9ktl"
Aug 04 18:01:45.722533 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:45.722396    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/726c7c38-d14e-4db0-b34d-d8ad5bb965b1-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-k9ktl\" (UID: \"726c7c38-d14e-4db0-b34d-d8ad5bb965b1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-k9ktl"
Aug 04 18:01:45.961035 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:45.961001    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-k9ktl"
Aug 04 18:01:46.116754 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:46.116722    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-k9ktl"]
Aug 04 18:01:46.120499 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:01:46.120014    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726c7c38_d14e_4db0_b34d_d8ad5bb965b1.slice/crio-6915802f0018112dba8e77655649b1135cb04ca800362310aa6e826b89b25e2e WatchSource:0}: Error finding container 6915802f0018112dba8e77655649b1135cb04ca800362310aa6e826b89b25e2e: Status 404 returned error can't find the container with id 6915802f0018112dba8e77655649b1135cb04ca800362310aa6e826b89b25e2e
Aug 04 18:01:47.035340 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:47.035288    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-k9ktl" event={"ID":"726c7c38-d14e-4db0-b34d-d8ad5bb965b1","Type":"ContainerStarted","Data":"6915802f0018112dba8e77655649b1135cb04ca800362310aa6e826b89b25e2e"}
Aug 04 18:01:47.999740 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:47.999662    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r9ntf"
Aug 04 18:01:49.044612 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:49.044574    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6944cbf757-l4rr2" event={"ID":"da619ae0-4438-48c9-b256-ef8d6c677b44","Type":"ContainerStarted","Data":"22605e034159b38e52fc25d84c2f6373d4f1006c1d618050c73fd16e1b5a684a"}
Aug 04 18:01:49.046552 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:49.046523    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-k9ktl" event={"ID":"726c7c38-d14e-4db0-b34d-d8ad5bb965b1","Type":"ContainerStarted","Data":"5586d904ece06b0433080cb42c57c30e88f6cf0c4aa8791d3e1c0cb29f4cb690"}
Aug 04 18:01:49.046679 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:49.046575    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-k9ktl"
Aug 04 18:01:49.052137 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:49.052118    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-k9ktl"
Aug 04 18:01:49.067382 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:49.067340    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6944cbf757-l4rr2" podStartSLOduration=1.5635212809999999 podStartE2EDuration="5.067322116s" totalImagesPullingTime="3.503800835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:49.066136927 +0000 UTC m=+112.036781291" watchObservedRunningTime="2026-08-04 18:01:49.067322116 +0000 UTC m=+112.037966459"
Aug 04 18:01:49.084934 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:49.084883    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-k9ktl" podStartSLOduration=1.9575405799999999 podStartE2EDuration="4.084863018s" totalImagesPullingTime="2.127322438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:49.084197023 +0000 UTC m=+112.054841363" watchObservedRunningTime="2026-08-04 18:01:49.084863018 +0000 UTC m=+112.055507360"
Aug 04 18:01:50.163610 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.163575    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-n25fm"]
Aug 04 18:01:50.166970 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.166952    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm"
Aug 04 18:01:50.169776 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.169755    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Aug 04 18:01:50.170132 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.170114    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Aug 04 18:01:50.170132 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.170124    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 04 18:01:50.170290 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.170192    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x7kbp\""
Aug 04 18:01:50.171118 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.171101    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 04 18:01:50.171351 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.171336    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 04 18:01:50.178678 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.178652    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-n25fm"]
Aug 04 18:01:50.261756 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.261720    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6f2r\" (UniqueName: \"kubernetes.io/projected/ef679628-6b96-4bce-9ee7-736b32f4d2f5-kube-api-access-w6f2r\") pod \"prometheus-operator-758d7cb874-n25fm\" (UID: \"ef679628-6b96-4bce-9ee7-736b32f4d2f5\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm"
Aug 04 18:01:50.261756 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.261760    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef679628-6b96-4bce-9ee7-736b32f4d2f5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-n25fm\" (UID: \"ef679628-6b96-4bce-9ee7-736b32f4d2f5\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm"
Aug 04 18:01:50.262018 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.261849    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef679628-6b96-4bce-9ee7-736b32f4d2f5-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-n25fm\" (UID: \"ef679628-6b96-4bce-9ee7-736b32f4d2f5\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm"
Aug 04 18:01:50.262018 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.261919    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef679628-6b96-4bce-9ee7-736b32f4d2f5-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-n25fm\" (UID: \"ef679628-6b96-4bce-9ee7-736b32f4d2f5\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm"
Aug 04 18:01:50.362943 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.362903    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6f2r\" (UniqueName: \"kubernetes.io/projected/ef679628-6b96-4bce-9ee7-736b32f4d2f5-kube-api-access-w6f2r\") pod \"prometheus-operator-758d7cb874-n25fm\" (UID: \"ef679628-6b96-4bce-9ee7-736b32f4d2f5\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm"
Aug 04 18:01:50.363098 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.362952    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef679628-6b96-4bce-9ee7-736b32f4d2f5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-n25fm\" (UID: \"ef679628-6b96-4bce-9ee7-736b32f4d2f5\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm"
Aug 04 18:01:50.363098 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.362995    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef679628-6b96-4bce-9ee7-736b32f4d2f5-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-n25fm\" (UID: \"ef679628-6b96-4bce-9ee7-736b32f4d2f5\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm"
Aug 04 18:01:50.363098 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.363029    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef679628-6b96-4bce-9ee7-736b32f4d2f5-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-n25fm\" (UID: \"ef679628-6b96-4bce-9ee7-736b32f4d2f5\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm"
Aug 04 18:01:50.363814 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.363787    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef679628-6b96-4bce-9ee7-736b32f4d2f5-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-n25fm\" (UID: \"ef679628-6b96-4bce-9ee7-736b32f4d2f5\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm"
Aug 04 18:01:50.365776 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.365748    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef679628-6b96-4bce-9ee7-736b32f4d2f5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-n25fm\" (UID: \"ef679628-6b96-4bce-9ee7-736b32f4d2f5\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm"
Aug 04 18:01:50.365776 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.365762    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef679628-6b96-4bce-9ee7-736b32f4d2f5-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-n25fm\" (UID: \"ef679628-6b96-4bce-9ee7-736b32f4d2f5\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm"
Aug 04 18:01:50.378170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.378144    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6f2r\" (UniqueName: \"kubernetes.io/projected/ef679628-6b96-4bce-9ee7-736b32f4d2f5-kube-api-access-w6f2r\") pod \"prometheus-operator-758d7cb874-n25fm\" (UID: \"ef679628-6b96-4bce-9ee7-736b32f4d2f5\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm"
Aug 04 18:01:50.480418 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:50.480332    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm"
Aug 04 18:01:54.605451 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:54.605417    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:54.605913 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:54.605483    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:54.611701 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:54.611679    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:55.072615 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:55.072523    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:01:56.241662 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:56.241624    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-n25fm"]
Aug 04 18:01:56.997176 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:56.997142    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:01:57.077602 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:57.077569    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-q6rn5" event={"ID":"3aa47b21-f214-40ef-a37d-29d68ba50805","Type":"ContainerStarted","Data":"641fbba14d929f166d9ab403b019ac57173cba550a38107ff51eb0c3ec35bec3"}
Aug 04 18:01:57.078176 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:57.078150    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-q6rn5"
Aug 04 18:01:57.079847 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:57.079821    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm" event={"ID":"ef679628-6b96-4bce-9ee7-736b32f4d2f5","Type":"ContainerStarted","Data":"8631bbc279c7871b96dffd844550f3032f95bc70ddf0b47ca1e814aab14aca16"}
Aug 04 18:01:57.090318 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:57.090295    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-q6rn5"
Aug 04 18:01:57.098438 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:57.098376    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-q6rn5" podStartSLOduration=1.662991957 podStartE2EDuration="18.098356065s" totalImagesPullingTime="16.435364108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:57.09786813 +0000 UTC m=+120.068512469" watchObservedRunningTime="2026-08-04 18:01:57.098356065 +0000 UTC m=+120.069000407"
Aug 04 18:01:59.660647 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:01:59.660616    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6944cbf757-l4rr2"]
Aug 04 18:02:00.094381 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:00.092512    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm" event={"ID":"ef679628-6b96-4bce-9ee7-736b32f4d2f5","Type":"ContainerStarted","Data":"14dfa1eadc92227daf8c362fae2d290ca883afc022039fa391e68103ddbc5381"}
Aug 04 18:02:00.094381 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:00.092584    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm" event={"ID":"ef679628-6b96-4bce-9ee7-736b32f4d2f5","Type":"ContainerStarted","Data":"c4240a98665157f4aa5badea72bddb3206b2e6d0ccf4434a98b7dae2f3da1e34"}
Aug 04 18:02:00.119458 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:00.119407    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-n25fm" podStartSLOduration=7.18887288 podStartE2EDuration="10.119386945s" totalImagesPullingTime="2.930514065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:00.118926636 +0000 UTC m=+123.089570999" watchObservedRunningTime="2026-08-04 18:02:00.119386945 +0000 UTC m=+123.090031287"
Aug 04 18:02:01.608967 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:01.608933    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54954fb884-m76mm"]
Aug 04 18:02:06.395514 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.395480    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs\") pod \"network-metrics-daemon-r27tq\" (UID: \"4fea01c7-a50c-40f7-a628-9745002f5158\") " pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:02:06.397747 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.397720    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fea01c7-a50c-40f7-a628-9745002f5158-metrics-certs\") pod \"network-metrics-daemon-r27tq\" (UID: \"4fea01c7-a50c-40f7-a628-9745002f5158\") " pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:02:06.469490 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.469458    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"]
Aug 04 18:02:06.539342 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.539293    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nv7dq"]
Aug 04 18:02:06.539487 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.539436    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.545040 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.545012    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-g45vx\""
Aug 04 18:02:06.545190 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.545051    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Aug 04 18:02:06.545190 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.545053    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Aug 04 18:02:06.545190 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.545132    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Aug 04 18:02:06.554179 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.554155    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"]
Aug 04 18:02:06.554338 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.554291    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.558658 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.558635    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 04 18:02:06.558909 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.558681    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6dtkz\""
Aug 04 18:02:06.558909 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.558823    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 04 18:02:06.559087 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.559072    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 04 18:02:06.597004 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.596973    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fc58f3af-bbf6-4904-a741-809c9dbeac97-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.597004 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.597005    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmlvc\" (UniqueName: \"kubernetes.io/projected/fc58f3af-bbf6-4904-a741-809c9dbeac97-kube-api-access-hmlvc\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.597205 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.597038    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc58f3af-bbf6-4904-a741-809c9dbeac97-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.597205 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.597055    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc58f3af-bbf6-4904-a741-809c9dbeac97-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.597205 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.597116    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fc58f3af-bbf6-4904-a741-809c9dbeac97-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.597205 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.597147    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc58f3af-bbf6-4904-a741-809c9dbeac97-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.686382 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.686288    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hp4qc\""
Aug 04 18:02:06.693969 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.693938    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r27tq"
Aug 04 18:02:06.698495 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.698465    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e591a246-2dbf-4e5a-bf50-14b1099c86ab-sys\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.698634 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.698525    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fc58f3af-bbf6-4904-a741-809c9dbeac97-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.698634 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.698558    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e591a246-2dbf-4e5a-bf50-14b1099c86ab-root\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.698634 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.698590    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc58f3af-bbf6-4904-a741-809c9dbeac97-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.698768 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.698641    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fc58f3af-bbf6-4904-a741-809c9dbeac97-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.698768 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.698693    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e591a246-2dbf-4e5a-bf50-14b1099c86ab-node-exporter-textfile\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.698867 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.698796    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e591a246-2dbf-4e5a-bf50-14b1099c86ab-node-exporter-wtmp\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.698867 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.698839    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fc58f3af-bbf6-4904-a741-809c9dbeac97-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.698867 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.698844    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmlvc\" (UniqueName: \"kubernetes.io/projected/fc58f3af-bbf6-4904-a741-809c9dbeac97-kube-api-access-hmlvc\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.699017 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.698884    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e591a246-2dbf-4e5a-bf50-14b1099c86ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.699017 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.698915    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e591a246-2dbf-4e5a-bf50-14b1099c86ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.699017 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.698942    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e591a246-2dbf-4e5a-bf50-14b1099c86ab-metrics-client-ca\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.699017 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.698970    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e591a246-2dbf-4e5a-bf50-14b1099c86ab-node-exporter-tls\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.699017 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.699000    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc58f3af-bbf6-4904-a741-809c9dbeac97-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.699017 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.699018    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc58f3af-bbf6-4904-a741-809c9dbeac97-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.699357 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.699043    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prvz2\" (UniqueName: \"kubernetes.io/projected/e591a246-2dbf-4e5a-bf50-14b1099c86ab-kube-api-access-prvz2\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.699478 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.699459    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fc58f3af-bbf6-4904-a741-809c9dbeac97-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.699764 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.699743    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc58f3af-bbf6-4904-a741-809c9dbeac97-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.701477 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.701444    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc58f3af-bbf6-4904-a741-809c9dbeac97-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.701648 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.701629    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc58f3af-bbf6-4904-a741-809c9dbeac97-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.708505 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.708479    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmlvc\" (UniqueName: \"kubernetes.io/projected/fc58f3af-bbf6-4904-a741-809c9dbeac97-kube-api-access-hmlvc\") pod \"kube-state-metrics-6bd9664c4-5vmkh\" (UID: \"fc58f3af-bbf6-4904-a741-809c9dbeac97\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.800142 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.800101    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e591a246-2dbf-4e5a-bf50-14b1099c86ab-root\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.800297 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.800179    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e591a246-2dbf-4e5a-bf50-14b1099c86ab-node-exporter-textfile\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.800297 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.800205    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e591a246-2dbf-4e5a-bf50-14b1099c86ab-node-exporter-wtmp\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.800297 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.800225    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e591a246-2dbf-4e5a-bf50-14b1099c86ab-root\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.800297 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.800245    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e591a246-2dbf-4e5a-bf50-14b1099c86ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.800297 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.800289    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e591a246-2dbf-4e5a-bf50-14b1099c86ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.800572 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.800345    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e591a246-2dbf-4e5a-bf50-14b1099c86ab-metrics-client-ca\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.800572 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.800373    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e591a246-2dbf-4e5a-bf50-14b1099c86ab-node-exporter-tls\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.800572 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.800388    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e591a246-2dbf-4e5a-bf50-14b1099c86ab-node-exporter-wtmp\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.800572 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.800405    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prvz2\" (UniqueName: \"kubernetes.io/projected/e591a246-2dbf-4e5a-bf50-14b1099c86ab-kube-api-access-prvz2\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.800572 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.800461    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e591a246-2dbf-4e5a-bf50-14b1099c86ab-sys\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.800572 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.800548    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e591a246-2dbf-4e5a-bf50-14b1099c86ab-sys\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.800846 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.800594    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e591a246-2dbf-4e5a-bf50-14b1099c86ab-node-exporter-textfile\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.801767 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.801722    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e591a246-2dbf-4e5a-bf50-14b1099c86ab-metrics-client-ca\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.802215 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.802138    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e591a246-2dbf-4e5a-bf50-14b1099c86ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.804076 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.804049    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e591a246-2dbf-4e5a-bf50-14b1099c86ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.804346 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.804328    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e591a246-2dbf-4e5a-bf50-14b1099c86ab-node-exporter-tls\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.809912 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.809892    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prvz2\" (UniqueName: \"kubernetes.io/projected/e591a246-2dbf-4e5a-bf50-14b1099c86ab-kube-api-access-prvz2\") pod \"node-exporter-nv7dq\" (UID: \"e591a246-2dbf-4e5a-bf50-14b1099c86ab\") " pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.846457 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.846435    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r27tq"]
Aug 04 18:02:06.848881 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:02:06.848854    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fea01c7_a50c_40f7_a628_9745002f5158.slice/crio-aa7e6c5b07bb1babcbaa505fbb31042199c6c85778c27a7a9a60fd990cd2b0c7 WatchSource:0}: Error finding container aa7e6c5b07bb1babcbaa505fbb31042199c6c85778c27a7a9a60fd990cd2b0c7: Status 404 returned error can't find the container with id aa7e6c5b07bb1babcbaa505fbb31042199c6c85778c27a7a9a60fd990cd2b0c7
Aug 04 18:02:06.848973 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.848931    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"
Aug 04 18:02:06.862109 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.862089    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nv7dq"
Aug 04 18:02:06.891792 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:02:06.891762    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode591a246_2dbf_4e5a_bf50_14b1099c86ab.slice/crio-b214536bc5bed617550b1524aeecf1588b4686cb3ffb16dbe20f9c61332a8a49 WatchSource:0}: Error finding container b214536bc5bed617550b1524aeecf1588b4686cb3ffb16dbe20f9c61332a8a49: Status 404 returned error can't find the container with id b214536bc5bed617550b1524aeecf1588b4686cb3ffb16dbe20f9c61332a8a49
Aug 04 18:02:06.998346 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:06.998301    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh"]
Aug 04 18:02:07.000534 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:02:07.000510    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc58f3af_bbf6_4904_a741_809c9dbeac97.slice/crio-0309ab9d91838f39f3cbb0e6319f357fc2d6275c754f25a5d123422bdc5bb109 WatchSource:0}: Error finding container 0309ab9d91838f39f3cbb0e6319f357fc2d6275c754f25a5d123422bdc5bb109: Status 404 returned error can't find the container with id 0309ab9d91838f39f3cbb0e6319f357fc2d6275c754f25a5d123422bdc5bb109
Aug 04 18:02:07.122595 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:07.122566    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nv7dq" event={"ID":"e591a246-2dbf-4e5a-bf50-14b1099c86ab","Type":"ContainerStarted","Data":"b214536bc5bed617550b1524aeecf1588b4686cb3ffb16dbe20f9c61332a8a49"}
Aug 04 18:02:07.123712 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:07.123691    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r27tq" event={"ID":"4fea01c7-a50c-40f7-a628-9745002f5158","Type":"ContainerStarted","Data":"aa7e6c5b07bb1babcbaa505fbb31042199c6c85778c27a7a9a60fd990cd2b0c7"}
Aug 04 18:02:07.124723 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:07.124699    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh" event={"ID":"fc58f3af-bbf6-4904-a741-809c9dbeac97","Type":"ContainerStarted","Data":"0309ab9d91838f39f3cbb0e6319f357fc2d6275c754f25a5d123422bdc5bb109"}
Aug 04 18:02:09.133746 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:09.133705    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r27tq" event={"ID":"4fea01c7-a50c-40f7-a628-9745002f5158","Type":"ContainerStarted","Data":"0c89be89125cf0832d9fd7fb0f57355e4ea04484dcd9d5662349c83a8c1fd3be"}
Aug 04 18:02:09.135396 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:09.135337    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nv7dq" event={"ID":"e591a246-2dbf-4e5a-bf50-14b1099c86ab","Type":"ContainerStarted","Data":"e90366c472aab6734ef2ef684f8614092d55b93cad35e45b6d9025f1516fa84e"}
Aug 04 18:02:10.140549 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.140521    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r27tq" event={"ID":"4fea01c7-a50c-40f7-a628-9745002f5158","Type":"ContainerStarted","Data":"e5278babf241046cee4c4fb9991260d3d727a1fffa489cbd72186b507814a16f"}
Aug 04 18:02:10.142792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.142765    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh" event={"ID":"fc58f3af-bbf6-4904-a741-809c9dbeac97","Type":"ContainerStarted","Data":"3bd585b74996ff49bf3daca24974fcdbd63bf111df2267f8b9d1221ce5847ca6"}
Aug 04 18:02:10.142921 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.142824    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh" event={"ID":"fc58f3af-bbf6-4904-a741-809c9dbeac97","Type":"ContainerStarted","Data":"b42571ce2625ebb382b45b3a9992157374f2919ef71a5ef3fa07789eba43d82a"}
Aug 04 18:02:10.142921 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.142852    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh" event={"ID":"fc58f3af-bbf6-4904-a741-809c9dbeac97","Type":"ContainerStarted","Data":"a744b8d96b7d5b35b0d7e0097a5b78dd3cb8dba2aece49e15447056241bc4c12"}
Aug 04 18:02:10.144116 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.144090    2543 generic.go:356] "Generic (PLEG): container finished" podID="e591a246-2dbf-4e5a-bf50-14b1099c86ab" containerID="e90366c472aab6734ef2ef684f8614092d55b93cad35e45b6d9025f1516fa84e" exitCode=0
Aug 04 18:02:10.144224 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.144143    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nv7dq" event={"ID":"e591a246-2dbf-4e5a-bf50-14b1099c86ab","Type":"ContainerDied","Data":"e90366c472aab6734ef2ef684f8614092d55b93cad35e45b6d9025f1516fa84e"}
Aug 04 18:02:10.163675 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.163638    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r27tq" podStartSLOduration=130.119116509 podStartE2EDuration="2m12.163625518s" totalImagesPullingTime="2.044509009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:10.16138609 +0000 UTC m=+133.132030430" watchObservedRunningTime="2026-08-04 18:02:10.163625518 +0000 UTC m=+133.134269857"
Aug 04 18:02:10.208821 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.208785    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-5vmkh" podStartSLOduration=1.981117867 podStartE2EDuration="4.208768259s" totalImagesPullingTime="2.227650392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:10.187215168 +0000 UTC m=+133.157859509" watchObservedRunningTime="2026-08-04 18:02:10.208768259 +0000 UTC m=+133.179412598"
Aug 04 18:02:10.834483 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.834449    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7cc78b8f7d-96w89"]
Aug 04 18:02:10.858847 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.858823    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cc78b8f7d-96w89"]
Aug 04 18:02:10.858966 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.858935    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:10.861923 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.861904    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Aug 04 18:02:10.863027 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.863000    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Aug 04 18:02:10.863139 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.863076    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fm0f27a21lp3a\""
Aug 04 18:02:10.863139 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.863131    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Aug 04 18:02:10.863253 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.863188    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8kt9w\""
Aug 04 18:02:10.863253 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.863201    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Aug 04 18:02:10.930840 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.930815    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nrk9\" (UniqueName: \"kubernetes.io/projected/bf48811a-ab2a-4931-bdb3-bd99d3378bad-kube-api-access-8nrk9\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:10.930942 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.930870    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bf48811a-ab2a-4931-bdb3-bd99d3378bad-secret-metrics-server-client-certs\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:10.930942 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.930915    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bf48811a-ab2a-4931-bdb3-bd99d3378bad-client-ca-bundle\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:10.931029 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.930965    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf48811a-ab2a-4931-bdb3-bd99d3378bad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:10.931029 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.930988    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bf48811a-ab2a-4931-bdb3-bd99d3378bad-audit-log\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:10.931029 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.931007    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bf48811a-ab2a-4931-bdb3-bd99d3378bad-metrics-server-audit-profiles\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:10.931134 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:10.931028    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bf48811a-ab2a-4931-bdb3-bd99d3378bad-secret-metrics-server-tls\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.032060 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.032036    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bf48811a-ab2a-4931-bdb3-bd99d3378bad-secret-metrics-server-client-certs\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.032197 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.032072    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bf48811a-ab2a-4931-bdb3-bd99d3378bad-client-ca-bundle\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.032197 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.032104    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf48811a-ab2a-4931-bdb3-bd99d3378bad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.032197 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.032132    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bf48811a-ab2a-4931-bdb3-bd99d3378bad-audit-log\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.032197 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.032159    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bf48811a-ab2a-4931-bdb3-bd99d3378bad-metrics-server-audit-profiles\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.032197 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.032189    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bf48811a-ab2a-4931-bdb3-bd99d3378bad-secret-metrics-server-tls\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.032520 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.032224    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nrk9\" (UniqueName: \"kubernetes.io/projected/bf48811a-ab2a-4931-bdb3-bd99d3378bad-kube-api-access-8nrk9\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.032770 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.032747    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bf48811a-ab2a-4931-bdb3-bd99d3378bad-audit-log\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.033083 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.033035    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf48811a-ab2a-4931-bdb3-bd99d3378bad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.033480 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.033458    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bf48811a-ab2a-4931-bdb3-bd99d3378bad-metrics-server-audit-profiles\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.034822 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.034800    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bf48811a-ab2a-4931-bdb3-bd99d3378bad-secret-metrics-server-tls\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.034932 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.034912    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bf48811a-ab2a-4931-bdb3-bd99d3378bad-secret-metrics-server-client-certs\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.034932 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.034921    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bf48811a-ab2a-4931-bdb3-bd99d3378bad-client-ca-bundle\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.048114 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.048090    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nrk9\" (UniqueName: \"kubernetes.io/projected/bf48811a-ab2a-4931-bdb3-bd99d3378bad-kube-api-access-8nrk9\") pod \"metrics-server-7cc78b8f7d-96w89\" (UID: \"bf48811a-ab2a-4931-bdb3-bd99d3378bad\") " pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.149936 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.149896    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nv7dq" event={"ID":"e591a246-2dbf-4e5a-bf50-14b1099c86ab","Type":"ContainerStarted","Data":"033b768736da6cb8f5048e89c7a71e8ff31f25a425bfa8121376898bffb25344"}
Aug 04 18:02:11.150252 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.149960    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nv7dq" event={"ID":"e591a246-2dbf-4e5a-bf50-14b1099c86ab","Type":"ContainerStarted","Data":"013ec1e119d633359459b1a049efad0eb08f26e61bca1d941cb44b5224c76adc"}
Aug 04 18:02:11.168126 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.168093    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:11.176799 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.176758    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nv7dq" podStartSLOduration=3.085135905 podStartE2EDuration="5.176745555s" totalImagesPullingTime="2.001672155s" totalInitContainerRuntime="89.937495ms" isStatefulPod=true podCreationTimestamp="2026-08-04 18:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:11.176217772 +0000 UTC m=+134.146862124" watchObservedRunningTime="2026-08-04 18:02:11.176745555 +0000 UTC m=+134.147389895"
Aug 04 18:02:11.333780 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.333757    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cc78b8f7d-96w89"]
Aug 04 18:02:11.336139 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:02:11.336112    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf48811a_ab2a_4931_bdb3_bd99d3378bad.slice/crio-62448e148c3c84c821dd2e754013e4d961d50835e61aaa959deb0595fa98349b WatchSource:0}: Error finding container 62448e148c3c84c821dd2e754013e4d961d50835e61aaa959deb0595fa98349b: Status 404 returned error can't find the container with id 62448e148c3c84c821dd2e754013e4d961d50835e61aaa959deb0595fa98349b
Aug 04 18:02:11.518302 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.518248    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b9d798b78-wm8gf"]
Aug 04 18:02:11.551162 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.551140    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.559624 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.559605    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Aug 04 18:02:11.564431 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.564408    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b9d798b78-wm8gf"]
Aug 04 18:02:11.637697 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.637672    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-oauth-config\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.637791 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.637703    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-config\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.637791 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.637731    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-serving-cert\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.637873 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.637807    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-oauth-serving-cert\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.637923 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.637887    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnd6p\" (UniqueName: \"kubernetes.io/projected/97ae62f9-30c8-494c-bc54-d188a6cd7cda-kube-api-access-rnd6p\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.637967 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.637928    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-service-ca\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.638018 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.637962    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-trusted-ca-bundle\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.738522 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.738501    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnd6p\" (UniqueName: \"kubernetes.io/projected/97ae62f9-30c8-494c-bc54-d188a6cd7cda-kube-api-access-rnd6p\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.738631 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.738536    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-service-ca\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.738631 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.738552    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-trusted-ca-bundle\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.738631 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.738574    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-oauth-config\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.738631 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.738593    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-config\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.738631 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.738619    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-serving-cert\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.738888 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.738651    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-oauth-serving-cert\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.739249 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.739225    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-service-ca\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.739435 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.739415    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-config\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.739435 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.739425    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-oauth-serving-cert\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.739575 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.739439    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-trusted-ca-bundle\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.741054 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.741029    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-oauth-config\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.741130 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.741116    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-serving-cert\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.753561 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.753544    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnd6p\" (UniqueName: \"kubernetes.io/projected/97ae62f9-30c8-494c-bc54-d188a6cd7cda-kube-api-access-rnd6p\") pod \"console-7b9d798b78-wm8gf\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") " pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:11.860278 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:11.860255    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:12.040686 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.040658    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b9d798b78-wm8gf"]
Aug 04 18:02:12.042490 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:02:12.042464    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ae62f9_30c8_494c_bc54_d188a6cd7cda.slice/crio-d8ea29c335902c2ffe14d0b68381832b4b23e5fd8806e3822497d176c60951bb WatchSource:0}: Error finding container d8ea29c335902c2ffe14d0b68381832b4b23e5fd8806e3822497d176c60951bb: Status 404 returned error can't find the container with id d8ea29c335902c2ffe14d0b68381832b4b23e5fd8806e3822497d176c60951bb
Aug 04 18:02:12.155869 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.155840    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b9d798b78-wm8gf" event={"ID":"97ae62f9-30c8-494c-bc54-d188a6cd7cda","Type":"ContainerStarted","Data":"76f44914b4d042441873e2b13865dd30fdb266b12d0722c1e859deecf3d0ce78"}
Aug 04 18:02:12.156207 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.155898    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b9d798b78-wm8gf" event={"ID":"97ae62f9-30c8-494c-bc54-d188a6cd7cda","Type":"ContainerStarted","Data":"d8ea29c335902c2ffe14d0b68381832b4b23e5fd8806e3822497d176c60951bb"}
Aug 04 18:02:12.157176 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.157145    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89" event={"ID":"bf48811a-ab2a-4931-bdb3-bd99d3378bad","Type":"ContainerStarted","Data":"62448e148c3c84c821dd2e754013e4d961d50835e61aaa959deb0595fa98349b"}
Aug 04 18:02:12.744076 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.744041    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:02:12.768039 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.768006    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.771546 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.771501    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:02:12.772733 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.772709    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Aug 04 18:02:12.772844 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.772820    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Aug 04 18:02:12.772913 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.772820    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a9j5o0n0el8hn\""
Aug 04 18:02:12.773572 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.773362    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Aug 04 18:02:12.773572 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.773390    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Aug 04 18:02:12.773731 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.773583    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Aug 04 18:02:12.774470 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.774302    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Aug 04 18:02:12.774470 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.774416    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Aug 04 18:02:12.774995 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.774972    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Aug 04 18:02:12.776114 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.776089    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Aug 04 18:02:12.776357 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.776340    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Aug 04 18:02:12.777419 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.776723    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n7k8m\""
Aug 04 18:02:12.777419 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.776876    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Aug 04 18:02:12.777419 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.777098    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Aug 04 18:02:12.777587 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.777571    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Aug 04 18:02:12.789140 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.789115    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Aug 04 18:02:12.848877 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.848847    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849034 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.848887    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849034 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.848917    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849034 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.848939    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849186 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849034    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849186 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849082    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849186 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849113    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0c3ff52-1d60-480c-a73d-f074d4eee205-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849291 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849194    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849291 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849230    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0c3ff52-1d60-480c-a73d-f074d4eee205-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849291 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849245    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849291 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849263    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849489 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849381    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0c3ff52-1d60-480c-a73d-f074d4eee205-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849489 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849415    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849489 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849439    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849489 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849463    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42bxl\" (UniqueName: \"kubernetes.io/projected/e0c3ff52-1d60-480c-a73d-f074d4eee205-kube-api-access-42bxl\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849665 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849493    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849665 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849516    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-config\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849665 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849537    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849665 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849562    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.849665 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.849585    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.950734    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.950778    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.950806    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.950831    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.950855    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.950880    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.950907    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0c3ff52-1d60-480c-a73d-f074d4eee205-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.950956    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.950988    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0c3ff52-1d60-480c-a73d-f074d4eee205-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.951008    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.951030    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.951072    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0c3ff52-1d60-480c-a73d-f074d4eee205-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.951091    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.951114    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.951133    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42bxl\" (UniqueName: \"kubernetes.io/projected/e0c3ff52-1d60-480c-a73d-f074d4eee205-kube-api-access-42bxl\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.951170 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.951169    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.952047 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.951206    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-config\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.952047 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.951231    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.952047 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.951258    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.952047 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.951282    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.952047 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.952023    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.952300 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.952023    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.952394 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.952352    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0c3ff52-1d60-480c-a73d-f074d4eee205-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.953690 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.952998    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.953690 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.953005    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.955614 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.954560    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.955614 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.955271    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.955784 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.955723    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0c3ff52-1d60-480c-a73d-f074d4eee205-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.956994 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.956546    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0c3ff52-1d60-480c-a73d-f074d4eee205-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.956994 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.956935    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.957735 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.957707    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.958109 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.958086    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.958531 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.958508    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0c3ff52-1d60-480c-a73d-f074d4eee205-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.959279 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.959237    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.959644 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.959613    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.959981 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.959941    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.960279 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.960202    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.960720 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.960680    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.961460 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.961424    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0c3ff52-1d60-480c-a73d-f074d4eee205-config\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:12.967324 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:12.967286    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42bxl\" (UniqueName: \"kubernetes.io/projected/e0c3ff52-1d60-480c-a73d-f074d4eee205-kube-api-access-42bxl\") pod \"prometheus-k8s-0\" (UID: \"e0c3ff52-1d60-480c-a73d-f074d4eee205\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:13.086648 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:13.086579    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:13.186534 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:13.186480    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b9d798b78-wm8gf" podStartSLOduration=2.186459186 podStartE2EDuration="2.186459186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:13.18388728 +0000 UTC m=+136.154531621" watchObservedRunningTime="2026-08-04 18:02:13.186459186 +0000 UTC m=+136.157103525"
Aug 04 18:02:13.297088 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:13.297067    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:02:13.299299 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:02:13.299273    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c3ff52_1d60_480c_a73d_f074d4eee205.slice/crio-f0cb8dbb4a78d954d07a1c7c72b841c63d36cc9314a5e17a3f39f95e2c2a4ba7 WatchSource:0}: Error finding container f0cb8dbb4a78d954d07a1c7c72b841c63d36cc9314a5e17a3f39f95e2c2a4ba7: Status 404 returned error can't find the container with id f0cb8dbb4a78d954d07a1c7c72b841c63d36cc9314a5e17a3f39f95e2c2a4ba7
Aug 04 18:02:14.166558 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:14.166525    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0c3ff52-1d60-480c-a73d-f074d4eee205","Type":"ContainerStarted","Data":"f0cb8dbb4a78d954d07a1c7c72b841c63d36cc9314a5e17a3f39f95e2c2a4ba7"}
Aug 04 18:02:14.167866 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:14.167844    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89" event={"ID":"bf48811a-ab2a-4931-bdb3-bd99d3378bad","Type":"ContainerStarted","Data":"f38f0a6cd094b72f7b3acc290ddd3cae08c5b69eab017867daa01cdc2aee429a"}
Aug 04 18:02:14.192577 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:14.192525    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89" podStartSLOduration=2.031259576 podStartE2EDuration="4.192509618s" totalImagesPullingTime="2.161250042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:14.191833238 +0000 UTC m=+137.162477589" watchObservedRunningTime="2026-08-04 18:02:14.192509618 +0000 UTC m=+137.163153957"
Aug 04 18:02:15.174669 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:15.174609    2543 generic.go:356] "Generic (PLEG): container finished" podID="e0c3ff52-1d60-480c-a73d-f074d4eee205" containerID="7c8b3011464e47d0fcf2f5df5fad5dee832dee827f7e874472ffbddfd920413f" exitCode=0
Aug 04 18:02:15.174773 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:15.174693    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0c3ff52-1d60-480c-a73d-f074d4eee205","Type":"ContainerDied","Data":"7c8b3011464e47d0fcf2f5df5fad5dee832dee827f7e874472ffbddfd920413f"}
Aug 04 18:02:17.916827 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:17.916787    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b9d798b78-wm8gf"]
Aug 04 18:02:19.195174 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:19.195094    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0c3ff52-1d60-480c-a73d-f074d4eee205","Type":"ContainerStarted","Data":"31f96a5f7b784563775190a9118b7d52a5820938367bb2ccbf8f0af096f99a52"}
Aug 04 18:02:19.195527 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:19.195181    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0c3ff52-1d60-480c-a73d-f074d4eee205","Type":"ContainerStarted","Data":"26062ac5e94cfc9572565688b42ce854023e1cf72caf0f9a0045a18af6e1b309"}
Aug 04 18:02:21.205436 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:21.205410    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0c3ff52-1d60-480c-a73d-f074d4eee205","Type":"ContainerStarted","Data":"f370c5717e442a02da0abae2627604a1e02f8ea1780e94ff9cd624ff11e74bc1"}
Aug 04 18:02:21.860394 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:21.860366    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:22.212947 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:22.212869    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0c3ff52-1d60-480c-a73d-f074d4eee205","Type":"ContainerStarted","Data":"13ca4294bceba5503df0a0293d85e1c6a23c283588762fc99477ba921a4193fb"}
Aug 04 18:02:22.212947 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:22.212933    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0c3ff52-1d60-480c-a73d-f074d4eee205","Type":"ContainerStarted","Data":"0e15636abee0a801eb9d9f36cde1beecf3accb23680647ebc911c975bd1e840f"}
Aug 04 18:02:22.213535 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:22.212959    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0c3ff52-1d60-480c-a73d-f074d4eee205","Type":"ContainerStarted","Data":"9e542f052beb66c9fc4d9c6b2c95d8249d64373481033b74afd069c5b7f1493b"}
Aug 04 18:02:22.247546 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:22.247493    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.060489193 podStartE2EDuration="10.247474449s" totalImagesPullingTime="7.168493808s" totalInitContainerRuntime="18.491448ms" isStatefulPod=false podCreationTimestamp="2026-08-04 18:02:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:22.245921044 +0000 UTC m=+145.216565393" watchObservedRunningTime="2026-08-04 18:02:22.247474449 +0000 UTC m=+145.218118790"
Aug 04 18:02:23.087366 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:23.087337    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:24.681388 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:24.681337    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6944cbf757-l4rr2" podUID="da619ae0-4438-48c9-b256-ef8d6c677b44" containerName="console" containerID="cri-o://22605e034159b38e52fc25d84c2f6373d4f1006c1d618050c73fd16e1b5a684a" gracePeriod=15
Aug 04 18:02:24.918173 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:24.918151    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:02:25.060998 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.060930    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-oauth-serving-cert\") pod \"da619ae0-4438-48c9-b256-ef8d6c677b44\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") "
Aug 04 18:02:25.060998 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.060971    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-console-config\") pod \"da619ae0-4438-48c9-b256-ef8d6c677b44\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") "
Aug 04 18:02:25.061185 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.061039    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da619ae0-4438-48c9-b256-ef8d6c677b44-console-oauth-config\") pod \"da619ae0-4438-48c9-b256-ef8d6c677b44\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") "
Aug 04 18:02:25.061185 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.061067    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-service-ca\") pod \"da619ae0-4438-48c9-b256-ef8d6c677b44\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") "
Aug 04 18:02:25.061185 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.061088    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da619ae0-4438-48c9-b256-ef8d6c677b44-console-serving-cert\") pod \"da619ae0-4438-48c9-b256-ef8d6c677b44\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") "
Aug 04 18:02:25.061185 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.061127    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2m2f\" (UniqueName: \"kubernetes.io/projected/da619ae0-4438-48c9-b256-ef8d6c677b44-kube-api-access-d2m2f\") pod \"da619ae0-4438-48c9-b256-ef8d6c677b44\" (UID: \"da619ae0-4438-48c9-b256-ef8d6c677b44\") "
Aug 04 18:02:25.061398 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.061360    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "da619ae0-4438-48c9-b256-ef8d6c677b44" (UID: "da619ae0-4438-48c9-b256-ef8d6c677b44"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:25.061684 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.061639    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-console-config" (OuterVolumeSpecName: "console-config") pod "da619ae0-4438-48c9-b256-ef8d6c677b44" (UID: "da619ae0-4438-48c9-b256-ef8d6c677b44"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:25.061684 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.061652    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-service-ca" (OuterVolumeSpecName: "service-ca") pod "da619ae0-4438-48c9-b256-ef8d6c677b44" (UID: "da619ae0-4438-48c9-b256-ef8d6c677b44"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:25.063708 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.063685    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da619ae0-4438-48c9-b256-ef8d6c677b44-kube-api-access-d2m2f" (OuterVolumeSpecName: "kube-api-access-d2m2f") pod "da619ae0-4438-48c9-b256-ef8d6c677b44" (UID: "da619ae0-4438-48c9-b256-ef8d6c677b44"). InnerVolumeSpecName "kube-api-access-d2m2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:25.064086 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.064063    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da619ae0-4438-48c9-b256-ef8d6c677b44-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "da619ae0-4438-48c9-b256-ef8d6c677b44" (UID: "da619ae0-4438-48c9-b256-ef8d6c677b44"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:25.064202 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.064179    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da619ae0-4438-48c9-b256-ef8d6c677b44-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "da619ae0-4438-48c9-b256-ef8d6c677b44" (UID: "da619ae0-4438-48c9-b256-ef8d6c677b44"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:25.162567 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.162542    2543 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da619ae0-4438-48c9-b256-ef8d6c677b44-console-oauth-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:25.162567 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.162563    2543 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-service-ca\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:25.162567 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.162573    2543 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da619ae0-4438-48c9-b256-ef8d6c677b44-console-serving-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:25.162756 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.162582    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2m2f\" (UniqueName: \"kubernetes.io/projected/da619ae0-4438-48c9-b256-ef8d6c677b44-kube-api-access-d2m2f\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:25.162756 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.162591    2543 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-oauth-serving-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:25.162756 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.162599    2543 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da619ae0-4438-48c9-b256-ef8d6c677b44-console-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:25.225447 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.225414    2543 generic.go:356] "Generic (PLEG): container finished" podID="da619ae0-4438-48c9-b256-ef8d6c677b44" containerID="22605e034159b38e52fc25d84c2f6373d4f1006c1d618050c73fd16e1b5a684a" exitCode=0
Aug 04 18:02:25.225571 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.225457    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6944cbf757-l4rr2"
Aug 04 18:02:25.225571 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.225503    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6944cbf757-l4rr2" event={"ID":"da619ae0-4438-48c9-b256-ef8d6c677b44","Type":"ContainerDied","Data":"22605e034159b38e52fc25d84c2f6373d4f1006c1d618050c73fd16e1b5a684a"}
Aug 04 18:02:25.225571 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.225545    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6944cbf757-l4rr2" event={"ID":"da619ae0-4438-48c9-b256-ef8d6c677b44","Type":"ContainerDied","Data":"6065522e3c05a1133659dcd72636c885b6eb4577a1b2b51cd497159f4c21e205"}
Aug 04 18:02:25.225571 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.225565    2543 scope.go:117] "RemoveContainer" containerID="22605e034159b38e52fc25d84c2f6373d4f1006c1d618050c73fd16e1b5a684a"
Aug 04 18:02:25.234060 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.234044    2543 scope.go:117] "RemoveContainer" containerID="22605e034159b38e52fc25d84c2f6373d4f1006c1d618050c73fd16e1b5a684a"
Aug 04 18:02:25.234321 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:02:25.234288    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22605e034159b38e52fc25d84c2f6373d4f1006c1d618050c73fd16e1b5a684a\": container with ID starting with 22605e034159b38e52fc25d84c2f6373d4f1006c1d618050c73fd16e1b5a684a not found: ID does not exist" containerID="22605e034159b38e52fc25d84c2f6373d4f1006c1d618050c73fd16e1b5a684a"
Aug 04 18:02:25.234383 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.234340    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22605e034159b38e52fc25d84c2f6373d4f1006c1d618050c73fd16e1b5a684a"} err="failed to get container status \"22605e034159b38e52fc25d84c2f6373d4f1006c1d618050c73fd16e1b5a684a\": rpc error: code = NotFound desc = could not find container \"22605e034159b38e52fc25d84c2f6373d4f1006c1d618050c73fd16e1b5a684a\": container with ID starting with 22605e034159b38e52fc25d84c2f6373d4f1006c1d618050c73fd16e1b5a684a not found: ID does not exist"
Aug 04 18:02:25.247892 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.247872    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6944cbf757-l4rr2"]
Aug 04 18:02:25.256682 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.256655    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6944cbf757-l4rr2"]
Aug 04 18:02:25.574098 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:25.574072    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da619ae0-4438-48c9-b256-ef8d6c677b44" path="/var/lib/kubelet/pods/da619ae0-4438-48c9-b256-ef8d6c677b44/volumes"
Aug 04 18:02:26.629366 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.629283    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54954fb884-m76mm" podUID="227c9ed0-d258-49e8-bbc5-01e00f6b1ede" containerName="registry" containerID="cri-o://a26bbeaf283b2e8e97d4d93c194e8fec5c40ad54dbede5373e27a5547f24277b" gracePeriod=30
Aug 04 18:02:26.867171 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.867144    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:02:26.978765 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.978713    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls\") pod \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") "
Aug 04 18:02:26.978765 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.978753    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtzdg\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-kube-api-access-dtzdg\") pod \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") "
Aug 04 18:02:26.978900 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.978776    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-installation-pull-secrets\") pod \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") "
Aug 04 18:02:26.978900 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.978806    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-ca-trust-extracted\") pod \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") "
Aug 04 18:02:26.978900 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.978887    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-image-registry-private-configuration\") pod \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") "
Aug 04 18:02:26.979052 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.978920    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-trusted-ca\") pod \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") "
Aug 04 18:02:26.979052 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.978962    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-certificates\") pod \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") "
Aug 04 18:02:26.979052 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.978988    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-bound-sa-token\") pod \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\" (UID: \"227c9ed0-d258-49e8-bbc5-01e00f6b1ede\") "
Aug 04 18:02:26.979383 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.979354    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "227c9ed0-d258-49e8-bbc5-01e00f6b1ede" (UID: "227c9ed0-d258-49e8-bbc5-01e00f6b1ede"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:26.979489 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.979420    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "227c9ed0-d258-49e8-bbc5-01e00f6b1ede" (UID: "227c9ed0-d258-49e8-bbc5-01e00f6b1ede"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:26.981203 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.981174    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "227c9ed0-d258-49e8-bbc5-01e00f6b1ede" (UID: "227c9ed0-d258-49e8-bbc5-01e00f6b1ede"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:26.981359 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.981336    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "227c9ed0-d258-49e8-bbc5-01e00f6b1ede" (UID: "227c9ed0-d258-49e8-bbc5-01e00f6b1ede"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:26.981519 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.981499    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "227c9ed0-d258-49e8-bbc5-01e00f6b1ede" (UID: "227c9ed0-d258-49e8-bbc5-01e00f6b1ede"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:26.981797 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.981775    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "227c9ed0-d258-49e8-bbc5-01e00f6b1ede" (UID: "227c9ed0-d258-49e8-bbc5-01e00f6b1ede"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:26.981884 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.981860    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-kube-api-access-dtzdg" (OuterVolumeSpecName: "kube-api-access-dtzdg") pod "227c9ed0-d258-49e8-bbc5-01e00f6b1ede" (UID: "227c9ed0-d258-49e8-bbc5-01e00f6b1ede"). InnerVolumeSpecName "kube-api-access-dtzdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:26.990126 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:26.990105    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "227c9ed0-d258-49e8-bbc5-01e00f6b1ede" (UID: "227c9ed0-d258-49e8-bbc5-01e00f6b1ede"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:02:27.079672 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.079650    2543 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-trusted-ca\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:27.079672 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.079673    2543 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-certificates\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:27.079780 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.079683    2543 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-bound-sa-token\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:27.079780 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.079693    2543 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-registry-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:27.079780 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.079700    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtzdg\" (UniqueName: \"kubernetes.io/projected/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-kube-api-access-dtzdg\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:27.079780 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.079709    2543 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-installation-pull-secrets\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:27.079780 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.079717    2543 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-ca-trust-extracted\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:27.079780 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.079727    2543 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/227c9ed0-d258-49e8-bbc5-01e00f6b1ede-image-registry-private-configuration\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:27.235657 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.235593    2543 generic.go:356] "Generic (PLEG): container finished" podID="227c9ed0-d258-49e8-bbc5-01e00f6b1ede" containerID="a26bbeaf283b2e8e97d4d93c194e8fec5c40ad54dbede5373e27a5547f24277b" exitCode=0
Aug 04 18:02:27.235657 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.235645    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54954fb884-m76mm"
Aug 04 18:02:27.235770 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.235691    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54954fb884-m76mm" event={"ID":"227c9ed0-d258-49e8-bbc5-01e00f6b1ede","Type":"ContainerDied","Data":"a26bbeaf283b2e8e97d4d93c194e8fec5c40ad54dbede5373e27a5547f24277b"}
Aug 04 18:02:27.235808 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.235773    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54954fb884-m76mm" event={"ID":"227c9ed0-d258-49e8-bbc5-01e00f6b1ede","Type":"ContainerDied","Data":"3b4412fcad34f4ec37b7b59aeafdcf58acef97e3b26f39cdeb8d268547d7cf2b"}
Aug 04 18:02:27.235808 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.235797    2543 scope.go:117] "RemoveContainer" containerID="a26bbeaf283b2e8e97d4d93c194e8fec5c40ad54dbede5373e27a5547f24277b"
Aug 04 18:02:27.246954 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.246931    2543 scope.go:117] "RemoveContainer" containerID="a26bbeaf283b2e8e97d4d93c194e8fec5c40ad54dbede5373e27a5547f24277b"
Aug 04 18:02:27.247540 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:02:27.247515    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a26bbeaf283b2e8e97d4d93c194e8fec5c40ad54dbede5373e27a5547f24277b\": container with ID starting with a26bbeaf283b2e8e97d4d93c194e8fec5c40ad54dbede5373e27a5547f24277b not found: ID does not exist" containerID="a26bbeaf283b2e8e97d4d93c194e8fec5c40ad54dbede5373e27a5547f24277b"
Aug 04 18:02:27.247639 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.247547    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26bbeaf283b2e8e97d4d93c194e8fec5c40ad54dbede5373e27a5547f24277b"} err="failed to get container status \"a26bbeaf283b2e8e97d4d93c194e8fec5c40ad54dbede5373e27a5547f24277b\": rpc error: code = NotFound desc = could not find container \"a26bbeaf283b2e8e97d4d93c194e8fec5c40ad54dbede5373e27a5547f24277b\": container with ID starting with a26bbeaf283b2e8e97d4d93c194e8fec5c40ad54dbede5373e27a5547f24277b not found: ID does not exist"
Aug 04 18:02:27.259159 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.259131    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54954fb884-m76mm"]
Aug 04 18:02:27.265571 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.265544    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54954fb884-m76mm"]
Aug 04 18:02:27.575204 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:27.575143    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="227c9ed0-d258-49e8-bbc5-01e00f6b1ede" path="/var/lib/kubelet/pods/227c9ed0-d258-49e8-bbc5-01e00f6b1ede/volumes"
Aug 04 18:02:31.168515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:31.168478    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:31.169017 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:31.168590    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:42.939391 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:42.939325    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b9d798b78-wm8gf" podUID="97ae62f9-30c8-494c-bc54-d188a6cd7cda" containerName="console" containerID="cri-o://76f44914b4d042441873e2b13865dd30fdb266b12d0722c1e859deecf3d0ce78" gracePeriod=15
Aug 04 18:02:43.214328 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.214229    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:43.304612 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.304575    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-oauth-serving-cert\") pod \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") "
Aug 04 18:02:43.304612 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.304611    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-service-ca\") pod \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") "
Aug 04 18:02:43.304833 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.304634    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-config\") pod \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") "
Aug 04 18:02:43.304833 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.304665    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-oauth-config\") pod \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") "
Aug 04 18:02:43.304833 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.304787    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-serving-cert\") pod \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") "
Aug 04 18:02:43.304833 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.304828    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnd6p\" (UniqueName: \"kubernetes.io/projected/97ae62f9-30c8-494c-bc54-d188a6cd7cda-kube-api-access-rnd6p\") pod \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") "
Aug 04 18:02:43.305015 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.304883    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-trusted-ca-bundle\") pod \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\" (UID: \"97ae62f9-30c8-494c-bc54-d188a6cd7cda\") "
Aug 04 18:02:43.305072 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.305034    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "97ae62f9-30c8-494c-bc54-d188a6cd7cda" (UID: "97ae62f9-30c8-494c-bc54-d188a6cd7cda"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:43.305131 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.305095    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-config" (OuterVolumeSpecName: "console-config") pod "97ae62f9-30c8-494c-bc54-d188a6cd7cda" (UID: "97ae62f9-30c8-494c-bc54-d188a6cd7cda"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:43.305131 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.305119    2543 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-oauth-serving-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:43.305451 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.305327    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "97ae62f9-30c8-494c-bc54-d188a6cd7cda" (UID: "97ae62f9-30c8-494c-bc54-d188a6cd7cda"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:43.305572 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.305536    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-service-ca" (OuterVolumeSpecName: "service-ca") pod "97ae62f9-30c8-494c-bc54-d188a6cd7cda" (UID: "97ae62f9-30c8-494c-bc54-d188a6cd7cda"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:43.306917 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.306896    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "97ae62f9-30c8-494c-bc54-d188a6cd7cda" (UID: "97ae62f9-30c8-494c-bc54-d188a6cd7cda"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:43.307016 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.306995    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ae62f9-30c8-494c-bc54-d188a6cd7cda-kube-api-access-rnd6p" (OuterVolumeSpecName: "kube-api-access-rnd6p") pod "97ae62f9-30c8-494c-bc54-d188a6cd7cda" (UID: "97ae62f9-30c8-494c-bc54-d188a6cd7cda"). InnerVolumeSpecName "kube-api-access-rnd6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:43.307061 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.307034    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "97ae62f9-30c8-494c-bc54-d188a6cd7cda" (UID: "97ae62f9-30c8-494c-bc54-d188a6cd7cda"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:43.314114 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.314091    2543 generic.go:356] "Generic (PLEG): container finished" podID="97ae62f9-30c8-494c-bc54-d188a6cd7cda" containerID="76f44914b4d042441873e2b13865dd30fdb266b12d0722c1e859deecf3d0ce78" exitCode=0
Aug 04 18:02:43.314210 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.314135    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b9d798b78-wm8gf"
Aug 04 18:02:43.314210 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.314161    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b9d798b78-wm8gf" event={"ID":"97ae62f9-30c8-494c-bc54-d188a6cd7cda","Type":"ContainerDied","Data":"76f44914b4d042441873e2b13865dd30fdb266b12d0722c1e859deecf3d0ce78"}
Aug 04 18:02:43.314296 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.314222    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b9d798b78-wm8gf" event={"ID":"97ae62f9-30c8-494c-bc54-d188a6cd7cda","Type":"ContainerDied","Data":"d8ea29c335902c2ffe14d0b68381832b4b23e5fd8806e3822497d176c60951bb"}
Aug 04 18:02:43.314296 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.314244    2543 scope.go:117] "RemoveContainer" containerID="76f44914b4d042441873e2b13865dd30fdb266b12d0722c1e859deecf3d0ce78"
Aug 04 18:02:43.322893 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.322835    2543 scope.go:117] "RemoveContainer" containerID="76f44914b4d042441873e2b13865dd30fdb266b12d0722c1e859deecf3d0ce78"
Aug 04 18:02:43.323222 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:02:43.323198    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f44914b4d042441873e2b13865dd30fdb266b12d0722c1e859deecf3d0ce78\": container with ID starting with 76f44914b4d042441873e2b13865dd30fdb266b12d0722c1e859deecf3d0ce78 not found: ID does not exist" containerID="76f44914b4d042441873e2b13865dd30fdb266b12d0722c1e859deecf3d0ce78"
Aug 04 18:02:43.323327 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.323229    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f44914b4d042441873e2b13865dd30fdb266b12d0722c1e859deecf3d0ce78"} err="failed to get container status \"76f44914b4d042441873e2b13865dd30fdb266b12d0722c1e859deecf3d0ce78\": rpc error: code = NotFound desc = could not find container \"76f44914b4d042441873e2b13865dd30fdb266b12d0722c1e859deecf3d0ce78\": container with ID starting with 76f44914b4d042441873e2b13865dd30fdb266b12d0722c1e859deecf3d0ce78 not found: ID does not exist"
Aug 04 18:02:43.339628 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.339598    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b9d798b78-wm8gf"]
Aug 04 18:02:43.348729 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.348698    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b9d798b78-wm8gf"]
Aug 04 18:02:43.405681 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.405647    2543 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-oauth-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:43.405681 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.405674    2543 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-serving-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:43.405681 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.405685    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnd6p\" (UniqueName: \"kubernetes.io/projected/97ae62f9-30c8-494c-bc54-d188a6cd7cda-kube-api-access-rnd6p\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:43.405943 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.405694    2543 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-trusted-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:43.405943 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.405704    2543 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-service-ca\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:43.405943 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.405712    2543 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97ae62f9-30c8-494c-bc54-d188a6cd7cda-console-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:02:43.573856 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:43.573777    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97ae62f9-30c8-494c-bc54-d188a6cd7cda" path="/var/lib/kubelet/pods/97ae62f9-30c8-494c-bc54-d188a6cd7cda/volumes"
Aug 04 18:02:44.320649 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:44.320611    2543 generic.go:356] "Generic (PLEG): container finished" podID="7b6607f0-aea9-4a84-a327-6794dd2b7dd5" containerID="0a0a458434b56aeceb924aa46ceabc993b4867e954f5b7f3de12d6dfdc6668b9" exitCode=0
Aug 04 18:02:44.321105 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:44.320681    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-cvxwr" event={"ID":"7b6607f0-aea9-4a84-a327-6794dd2b7dd5","Type":"ContainerDied","Data":"0a0a458434b56aeceb924aa46ceabc993b4867e954f5b7f3de12d6dfdc6668b9"}
Aug 04 18:02:44.321168 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:44.321120    2543 scope.go:117] "RemoveContainer" containerID="0a0a458434b56aeceb924aa46ceabc993b4867e954f5b7f3de12d6dfdc6668b9"
Aug 04 18:02:44.322121 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:44.322099    2543 generic.go:356] "Generic (PLEG): container finished" podID="9e3c4757-7362-4733-a723-36271decfb8c" containerID="673d2d3b5a3c2663e5972d7cb9082a08e8c9e61f057bf44d41fd15258d55c1ac" exitCode=0
Aug 04 18:02:44.322222 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:44.322140    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m" event={"ID":"9e3c4757-7362-4733-a723-36271decfb8c","Type":"ContainerDied","Data":"673d2d3b5a3c2663e5972d7cb9082a08e8c9e61f057bf44d41fd15258d55c1ac"}
Aug 04 18:02:44.322472 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:44.322414    2543 scope.go:117] "RemoveContainer" containerID="673d2d3b5a3c2663e5972d7cb9082a08e8c9e61f057bf44d41fd15258d55c1ac"
Aug 04 18:02:45.328968 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:45.328901    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-cvxwr" event={"ID":"7b6607f0-aea9-4a84-a327-6794dd2b7dd5","Type":"ContainerStarted","Data":"da0f3d6d0d391f3336d8775bab60ca0e26cbe75db25ac77d89edfea3064a89c4"}
Aug 04 18:02:45.331720 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:45.331211    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-kgw2m" event={"ID":"9e3c4757-7362-4733-a723-36271decfb8c","Type":"ContainerStarted","Data":"354a93d63578d0593198406922eed56550e692070541895472b264caae7dc198"}
Aug 04 18:02:51.175595 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:51.175513    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:02:51.180149 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:02:51.180122    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7cc78b8f7d-96w89"
Aug 04 18:03:13.087184 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:03:13.087132    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:03:13.107343 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:03:13.107300    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:03:13.464095 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:03:13.464065    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:04:57.442390 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:04:57.442359    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/1.log"
Aug 04 18:04:57.447903 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:04:57.447881    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/1.log"
Aug 04 18:04:57.453409 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:04:57.453390    2543 kubelet.go:1648] "Image garbage collection succeeded"
Aug 04 18:08:43.905162 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.905117    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-8588677cfb-rf2zl"]
Aug 04 18:08:43.905610 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.905452    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ae62f9-30c8-494c-bc54-d188a6cd7cda" containerName="console"
Aug 04 18:08:43.905610 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.905464    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ae62f9-30c8-494c-bc54-d188a6cd7cda" containerName="console"
Aug 04 18:08:43.905610 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.905474    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="227c9ed0-d258-49e8-bbc5-01e00f6b1ede" containerName="registry"
Aug 04 18:08:43.905610 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.905479    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="227c9ed0-d258-49e8-bbc5-01e00f6b1ede" containerName="registry"
Aug 04 18:08:43.905610 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.905486    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da619ae0-4438-48c9-b256-ef8d6c677b44" containerName="console"
Aug 04 18:08:43.905610 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.905492    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="da619ae0-4438-48c9-b256-ef8d6c677b44" containerName="console"
Aug 04 18:08:43.905610 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.905545    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ae62f9-30c8-494c-bc54-d188a6cd7cda" containerName="console"
Aug 04 18:08:43.905610 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.905554    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="227c9ed0-d258-49e8-bbc5-01e00f6b1ede" containerName="registry"
Aug 04 18:08:43.905610 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.905559    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="da619ae0-4438-48c9-b256-ef8d6c677b44" containerName="console"
Aug 04 18:08:43.907552 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.907532    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-rf2zl"
Aug 04 18:08:43.910695 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.910662    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Aug 04 18:08:43.910889 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.910862    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Aug 04 18:08:43.911051 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.910922    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Aug 04 18:08:43.911142 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.910798    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Aug 04 18:08:43.912428 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.912390    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-hnfv4\""
Aug 04 18:08:43.919940 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:43.919911    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-rf2zl"]
Aug 04 18:08:44.005800 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:44.005761    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8ff864e-167e-48eb-a074-3253b1581e79-cert\") pod \"odh-model-controller-8588677cfb-rf2zl\" (UID: \"b8ff864e-167e-48eb-a074-3253b1581e79\") " pod="kserve/odh-model-controller-8588677cfb-rf2zl"
Aug 04 18:08:44.005994 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:44.005807    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ff864e-167e-48eb-a074-3253b1581e79-metrics-certs\") pod \"odh-model-controller-8588677cfb-rf2zl\" (UID: \"b8ff864e-167e-48eb-a074-3253b1581e79\") " pod="kserve/odh-model-controller-8588677cfb-rf2zl"
Aug 04 18:08:44.005994 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:44.005830    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npdb7\" (UniqueName: \"kubernetes.io/projected/b8ff864e-167e-48eb-a074-3253b1581e79-kube-api-access-npdb7\") pod \"odh-model-controller-8588677cfb-rf2zl\" (UID: \"b8ff864e-167e-48eb-a074-3253b1581e79\") " pod="kserve/odh-model-controller-8588677cfb-rf2zl"
Aug 04 18:08:44.106533 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:44.106497    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8ff864e-167e-48eb-a074-3253b1581e79-cert\") pod \"odh-model-controller-8588677cfb-rf2zl\" (UID: \"b8ff864e-167e-48eb-a074-3253b1581e79\") " pod="kserve/odh-model-controller-8588677cfb-rf2zl"
Aug 04 18:08:44.106533 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:44.106536    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ff864e-167e-48eb-a074-3253b1581e79-metrics-certs\") pod \"odh-model-controller-8588677cfb-rf2zl\" (UID: \"b8ff864e-167e-48eb-a074-3253b1581e79\") " pod="kserve/odh-model-controller-8588677cfb-rf2zl"
Aug 04 18:08:44.106812 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:44.106558    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npdb7\" (UniqueName: \"kubernetes.io/projected/b8ff864e-167e-48eb-a074-3253b1581e79-kube-api-access-npdb7\") pod \"odh-model-controller-8588677cfb-rf2zl\" (UID: \"b8ff864e-167e-48eb-a074-3253b1581e79\") " pod="kserve/odh-model-controller-8588677cfb-rf2zl"
Aug 04 18:08:44.106812 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:08:44.106678    2543 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Aug 04 18:08:44.106812 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:08:44.106776    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8ff864e-167e-48eb-a074-3253b1581e79-cert podName:b8ff864e-167e-48eb-a074-3253b1581e79 nodeName:}" failed. No retries permitted until 2026-08-04 18:08:44.606752267 +0000 UTC m=+527.577396599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b8ff864e-167e-48eb-a074-3253b1581e79-cert") pod "odh-model-controller-8588677cfb-rf2zl" (UID: "b8ff864e-167e-48eb-a074-3253b1581e79") : secret "odh-model-controller-webhook-cert" not found
Aug 04 18:08:44.106812 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:08:44.106686    2543 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Aug 04 18:08:44.107021 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:08:44.106818    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8ff864e-167e-48eb-a074-3253b1581e79-metrics-certs podName:b8ff864e-167e-48eb-a074-3253b1581e79 nodeName:}" failed. No retries permitted until 2026-08-04 18:08:44.60680867 +0000 UTC m=+527.577452989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8ff864e-167e-48eb-a074-3253b1581e79-metrics-certs") pod "odh-model-controller-8588677cfb-rf2zl" (UID: "b8ff864e-167e-48eb-a074-3253b1581e79") : secret "odh-model-controller-metrics-tls" not found
Aug 04 18:08:44.119452 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:44.119421    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npdb7\" (UniqueName: \"kubernetes.io/projected/b8ff864e-167e-48eb-a074-3253b1581e79-kube-api-access-npdb7\") pod \"odh-model-controller-8588677cfb-rf2zl\" (UID: \"b8ff864e-167e-48eb-a074-3253b1581e79\") " pod="kserve/odh-model-controller-8588677cfb-rf2zl"
Aug 04 18:08:44.610702 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:44.610659    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8ff864e-167e-48eb-a074-3253b1581e79-cert\") pod \"odh-model-controller-8588677cfb-rf2zl\" (UID: \"b8ff864e-167e-48eb-a074-3253b1581e79\") " pod="kserve/odh-model-controller-8588677cfb-rf2zl"
Aug 04 18:08:44.610702 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:44.610706    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ff864e-167e-48eb-a074-3253b1581e79-metrics-certs\") pod \"odh-model-controller-8588677cfb-rf2zl\" (UID: \"b8ff864e-167e-48eb-a074-3253b1581e79\") " pod="kserve/odh-model-controller-8588677cfb-rf2zl"
Aug 04 18:08:44.613394 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:44.613353    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8ff864e-167e-48eb-a074-3253b1581e79-cert\") pod \"odh-model-controller-8588677cfb-rf2zl\" (UID: \"b8ff864e-167e-48eb-a074-3253b1581e79\") " pod="kserve/odh-model-controller-8588677cfb-rf2zl"
Aug 04 18:08:44.613394 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:44.613353    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ff864e-167e-48eb-a074-3253b1581e79-metrics-certs\") pod \"odh-model-controller-8588677cfb-rf2zl\" (UID: \"b8ff864e-167e-48eb-a074-3253b1581e79\") " pod="kserve/odh-model-controller-8588677cfb-rf2zl"
Aug 04 18:08:44.821304 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:44.821259    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-rf2zl"
Aug 04 18:08:44.954520 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:44.954486    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-rf2zl"]
Aug 04 18:08:44.957813 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:08:44.957786    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ff864e_167e_48eb_a074_3253b1581e79.slice/crio-94797cf3a818e47a58a62246e2a41e9fa4fe4be783e64a485363bdf82eab35ac WatchSource:0}: Error finding container 94797cf3a818e47a58a62246e2a41e9fa4fe4be783e64a485363bdf82eab35ac: Status 404 returned error can't find the container with id 94797cf3a818e47a58a62246e2a41e9fa4fe4be783e64a485363bdf82eab35ac
Aug 04 18:08:44.959123 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:44.959106    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:08:45.691764 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:45.691722    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-rf2zl" event={"ID":"b8ff864e-167e-48eb-a074-3253b1581e79","Type":"ContainerStarted","Data":"94797cf3a818e47a58a62246e2a41e9fa4fe4be783e64a485363bdf82eab35ac"}
Aug 04 18:08:48.706436 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:48.706395    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-rf2zl" event={"ID":"b8ff864e-167e-48eb-a074-3253b1581e79","Type":"ContainerStarted","Data":"e40f01755082d6a61242a7301c9b2f321aef11883eb405c1fde6264e93ce8c41"}
Aug 04 18:08:48.706807 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:48.706539    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-8588677cfb-rf2zl"
Aug 04 18:08:48.724749 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:48.724692    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-8588677cfb-rf2zl" podStartSLOduration=2.991665855 podStartE2EDuration="5.724675444s" totalImagesPullingTime="2.733009589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:08:48.724382755 +0000 UTC m=+531.695027114" watchObservedRunningTime="2026-08-04 18:08:48.724675444 +0000 UTC m=+531.695319785"
Aug 04 18:08:59.712997 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:08:59.712915    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-8588677cfb-rf2zl"
Aug 04 18:09:00.530475 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:00.530444    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-2xwsw"]
Aug 04 18:09:00.535886 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:00.535863    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2xwsw"
Aug 04 18:09:00.538659 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:00.538631    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6slp5\""
Aug 04 18:09:00.538795 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:00.538682    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Aug 04 18:09:00.542100 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:00.542047    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2xwsw"]
Aug 04 18:09:00.639281 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:00.639244    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msqnk\" (UniqueName: \"kubernetes.io/projected/f9a69cf6-a3e9-431a-bcdc-9373c002910e-kube-api-access-msqnk\") pod \"s3-init-2xwsw\" (UID: \"f9a69cf6-a3e9-431a-bcdc-9373c002910e\") " pod="kserve/s3-init-2xwsw"
Aug 04 18:09:00.639281 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:00.639292    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a69cf6-a3e9-431a-bcdc-9373c002910e-model-cache\") pod \"s3-init-2xwsw\" (UID: \"f9a69cf6-a3e9-431a-bcdc-9373c002910e\") " pod="kserve/s3-init-2xwsw"
Aug 04 18:09:00.739818 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:00.739781    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a69cf6-a3e9-431a-bcdc-9373c002910e-model-cache\") pod \"s3-init-2xwsw\" (UID: \"f9a69cf6-a3e9-431a-bcdc-9373c002910e\") " pod="kserve/s3-init-2xwsw"
Aug 04 18:09:00.740206 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:00.739870    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msqnk\" (UniqueName: \"kubernetes.io/projected/f9a69cf6-a3e9-431a-bcdc-9373c002910e-kube-api-access-msqnk\") pod \"s3-init-2xwsw\" (UID: \"f9a69cf6-a3e9-431a-bcdc-9373c002910e\") " pod="kserve/s3-init-2xwsw"
Aug 04 18:09:00.740206 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:00.740183    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a69cf6-a3e9-431a-bcdc-9373c002910e-model-cache\") pod \"s3-init-2xwsw\" (UID: \"f9a69cf6-a3e9-431a-bcdc-9373c002910e\") " pod="kserve/s3-init-2xwsw"
Aug 04 18:09:00.752701 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:00.752674    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msqnk\" (UniqueName: \"kubernetes.io/projected/f9a69cf6-a3e9-431a-bcdc-9373c002910e-kube-api-access-msqnk\") pod \"s3-init-2xwsw\" (UID: \"f9a69cf6-a3e9-431a-bcdc-9373c002910e\") " pod="kserve/s3-init-2xwsw"
Aug 04 18:09:00.846094 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:00.846061    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2xwsw"
Aug 04 18:09:00.979217 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:00.979178    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2xwsw"]
Aug 04 18:09:00.982016 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:09:00.981988    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9a69cf6_a3e9_431a_bcdc_9373c002910e.slice/crio-0bd50c81f9e7362c461a2d1aa62e87a3cdcbdbc16a2fb19d99bbec5b9cfa83ea WatchSource:0}: Error finding container 0bd50c81f9e7362c461a2d1aa62e87a3cdcbdbc16a2fb19d99bbec5b9cfa83ea: Status 404 returned error can't find the container with id 0bd50c81f9e7362c461a2d1aa62e87a3cdcbdbc16a2fb19d99bbec5b9cfa83ea
Aug 04 18:09:01.757525 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:01.757480    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2xwsw" event={"ID":"f9a69cf6-a3e9-431a-bcdc-9373c002910e","Type":"ContainerStarted","Data":"0bd50c81f9e7362c461a2d1aa62e87a3cdcbdbc16a2fb19d99bbec5b9cfa83ea"}
Aug 04 18:09:05.777575 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:05.777536    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2xwsw" event={"ID":"f9a69cf6-a3e9-431a-bcdc-9373c002910e","Type":"ContainerStarted","Data":"8e3dd8fbf7af091f62880f968e02e8c9b0ad6a498f68a2de878e05d68ccb208d"}
Aug 04 18:09:06.782321 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:06.782272    2543 generic.go:356] "Generic (PLEG): container finished" podID="f9a69cf6-a3e9-431a-bcdc-9373c002910e" containerID="8e3dd8fbf7af091f62880f968e02e8c9b0ad6a498f68a2de878e05d68ccb208d" exitCode=0
Aug 04 18:09:06.782745 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:06.782372    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2xwsw" event={"ID":"f9a69cf6-a3e9-431a-bcdc-9373c002910e","Type":"ContainerDied","Data":"8e3dd8fbf7af091f62880f968e02e8c9b0ad6a498f68a2de878e05d68ccb208d"}
Aug 04 18:09:11.808409 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:11.808366    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2xwsw" event={"ID":"f9a69cf6-a3e9-431a-bcdc-9373c002910e","Type":"ContainerStarted","Data":"184db8b1e02028a03495751f66b03edb0a34d24470ec6596a7c8a5de39a1f3d1"}
Aug 04 18:09:11.828160 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:11.828109    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-2xwsw" podStartSLOduration=1.712993644 podStartE2EDuration="11.828090479s" totalImagesPullingTime="8.580513911s" totalInitContainerRuntime="1.534582924s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:09:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:09:11.827281257 +0000 UTC m=+554.797925610" watchObservedRunningTime="2026-08-04 18:09:11.828090479 +0000 UTC m=+554.798734820"
Aug 04 18:09:57.469117 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:57.469089    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/1.log"
Aug 04 18:09:57.472200 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:09:57.472175    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/1.log"
Aug 04 18:10:17.068704 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:17.068669    2543 generic.go:356] "Generic (PLEG): container finished" podID="f9a69cf6-a3e9-431a-bcdc-9373c002910e" containerID="184db8b1e02028a03495751f66b03edb0a34d24470ec6596a7c8a5de39a1f3d1" exitCode=0
Aug 04 18:10:17.069082 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:17.068749    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2xwsw" event={"ID":"f9a69cf6-a3e9-431a-bcdc-9373c002910e","Type":"ContainerDied","Data":"184db8b1e02028a03495751f66b03edb0a34d24470ec6596a7c8a5de39a1f3d1"}
Aug 04 18:10:18.209189 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:18.209155    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2xwsw"
Aug 04 18:10:18.256027 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:18.255993    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a69cf6-a3e9-431a-bcdc-9373c002910e-model-cache\") pod \"f9a69cf6-a3e9-431a-bcdc-9373c002910e\" (UID: \"f9a69cf6-a3e9-431a-bcdc-9373c002910e\") "
Aug 04 18:10:18.256241 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:18.256121    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msqnk\" (UniqueName: \"kubernetes.io/projected/f9a69cf6-a3e9-431a-bcdc-9373c002910e-kube-api-access-msqnk\") pod \"f9a69cf6-a3e9-431a-bcdc-9373c002910e\" (UID: \"f9a69cf6-a3e9-431a-bcdc-9373c002910e\") "
Aug 04 18:10:18.258408 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:18.258372    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a69cf6-a3e9-431a-bcdc-9373c002910e-kube-api-access-msqnk" (OuterVolumeSpecName: "kube-api-access-msqnk") pod "f9a69cf6-a3e9-431a-bcdc-9373c002910e" (UID: "f9a69cf6-a3e9-431a-bcdc-9373c002910e"). InnerVolumeSpecName "kube-api-access-msqnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:10:18.279194 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:18.279140    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a69cf6-a3e9-431a-bcdc-9373c002910e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9a69cf6-a3e9-431a-bcdc-9373c002910e" (UID: "f9a69cf6-a3e9-431a-bcdc-9373c002910e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:10:18.357483 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:18.357450    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msqnk\" (UniqueName: \"kubernetes.io/projected/f9a69cf6-a3e9-431a-bcdc-9373c002910e-kube-api-access-msqnk\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:10:18.357483 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:18.357479    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a69cf6-a3e9-431a-bcdc-9373c002910e-model-cache\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:10:19.078430 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:19.078399    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2xwsw"
Aug 04 18:10:19.078600 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:19.078403    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2xwsw" event={"ID":"f9a69cf6-a3e9-431a-bcdc-9373c002910e","Type":"ContainerDied","Data":"0bd50c81f9e7362c461a2d1aa62e87a3cdcbdbc16a2fb19d99bbec5b9cfa83ea"}
Aug 04 18:10:19.078600 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:19.078521    2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bd50c81f9e7362c461a2d1aa62e87a3cdcbdbc16a2fb19d99bbec5b9cfa83ea"
Aug 04 18:10:30.691722 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.691685    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"]
Aug 04 18:10:30.692213 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.691984    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a69cf6-a3e9-431a-bcdc-9373c002910e" containerName="download-hf-model"
Aug 04 18:10:30.692213 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.691996    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a69cf6-a3e9-431a-bcdc-9373c002910e" containerName="download-hf-model"
Aug 04 18:10:30.692213 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.692004    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a69cf6-a3e9-431a-bcdc-9373c002910e" containerName="s3-init"
Aug 04 18:10:30.692213 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.692009    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a69cf6-a3e9-431a-bcdc-9373c002910e" containerName="s3-init"
Aug 04 18:10:30.692213 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.692079    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9a69cf6-a3e9-431a-bcdc-9373c002910e" containerName="s3-init"
Aug 04 18:10:30.695646 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.695621    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.698274 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.698247    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-5a068-kube-rbac-proxy-sar-config\""
Aug 04 18:10:30.698467 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.698247    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:10:30.698467 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.698247    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:10:30.698591 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.698248    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:10:30.698780 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.698763    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:10:30.698866 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.698797    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-5a068-predictor-serving-cert\""
Aug 04 18:10:30.699525 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.699509    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vsp4f\""
Aug 04 18:10:30.709054 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.709024    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"]
Aug 04 18:10:30.768485 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.768446    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/936d63e2-5a19-4682-ac7e-6797a64783de-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.768697 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.768533    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/936d63e2-5a19-4682-ac7e-6797a64783de-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.768697 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.768564    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/936d63e2-5a19-4682-ac7e-6797a64783de-proxy-tls\") pod \"isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.768697 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.768599    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz68h\" (UniqueName: \"kubernetes.io/projected/936d63e2-5a19-4682-ac7e-6797a64783de-kube-api-access-tz68h\") pod \"isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.768697 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.768628    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/936d63e2-5a19-4682-ac7e-6797a64783de-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.808683 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.808644    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"]
Aug 04 18:10:30.812224 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.812202    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:30.818674 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.818646    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Aug 04 18:10:30.818674 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.818647    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Aug 04 18:10:30.830398 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.830357    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"]
Aug 04 18:10:30.869082 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.869043    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/936d63e2-5a19-4682-ac7e-6797a64783de-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.869082 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.869084    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/936d63e2-5a19-4682-ac7e-6797a64783de-proxy-tls\") pod \"isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.869405 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.869123    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz68h\" (UniqueName: \"kubernetes.io/projected/936d63e2-5a19-4682-ac7e-6797a64783de-kube-api-access-tz68h\") pod \"isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.869405 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.869154    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/936d63e2-5a19-4682-ac7e-6797a64783de-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.869405 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.869221    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/936d63e2-5a19-4682-ac7e-6797a64783de-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.869755 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.869734    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/936d63e2-5a19-4682-ac7e-6797a64783de-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.869932 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.869909    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/936d63e2-5a19-4682-ac7e-6797a64783de-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.869989 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.869924    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/936d63e2-5a19-4682-ac7e-6797a64783de-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.871811 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.871791    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/936d63e2-5a19-4682-ac7e-6797a64783de-proxy-tls\") pod \"isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.882698 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.882668    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz68h\" (UniqueName: \"kubernetes.io/projected/936d63e2-5a19-4682-ac7e-6797a64783de-kube-api-access-tz68h\") pod \"isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:30.891811 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.891773    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"]
Aug 04 18:10:30.896039 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.896014    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:30.898711 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.898679    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Aug 04 18:10:30.905966 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.905936    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"]
Aug 04 18:10:30.970804 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.970714    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/754f4c06-cff3-4f1d-816f-2276676045ca-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:30.970804 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.970769    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/754f4c06-cff3-4f1d-816f-2276676045ca-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:30.970997 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.970818    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/754f4c06-cff3-4f1d-816f-2276676045ca-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:30.970997 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.970857    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bc20656-1ded-4021-bd73-d6dcd469f19d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:30.970997 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.970885    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bc20656-1ded-4021-bd73-d6dcd469f19d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:30.970997 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.970914    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bc20656-1ded-4021-bd73-d6dcd469f19d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:30.970997 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.970948    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bc20656-1ded-4021-bd73-d6dcd469f19d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:30.970997 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.970992    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb7gw\" (UniqueName: \"kubernetes.io/projected/8bc20656-1ded-4021-bd73-d6dcd469f19d-kube-api-access-rb7gw\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:30.971178 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.971037    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qsfm\" (UniqueName: \"kubernetes.io/projected/754f4c06-cff3-4f1d-816f-2276676045ca-kube-api-access-8qsfm\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:30.971178 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:30.971075    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/754f4c06-cff3-4f1d-816f-2276676045ca-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:31.007116 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.007074    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:10:31.072473 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.072431    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/754f4c06-cff3-4f1d-816f-2276676045ca-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:31.072622 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.072512    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/754f4c06-cff3-4f1d-816f-2276676045ca-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:31.072622 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.072540    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/754f4c06-cff3-4f1d-816f-2276676045ca-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:31.072622 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.072568    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/754f4c06-cff3-4f1d-816f-2276676045ca-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:31.072622 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.072594    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bc20656-1ded-4021-bd73-d6dcd469f19d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:31.072622 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:10:31.072603    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Aug 04 18:10:31.072779 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.072624    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bc20656-1ded-4021-bd73-d6dcd469f19d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:31.072779 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.072654    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bc20656-1ded-4021-bd73-d6dcd469f19d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:31.072779 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:10:31.072703    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/754f4c06-cff3-4f1d-816f-2276676045ca-proxy-tls podName:754f4c06-cff3-4f1d-816f-2276676045ca nodeName:}" failed. No retries permitted until 2026-08-04 18:10:31.572675414 +0000 UTC m=+634.543319739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/754f4c06-cff3-4f1d-816f-2276676045ca-proxy-tls") pod "isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" (UID: "754f4c06-cff3-4f1d-816f-2276676045ca") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Aug 04 18:10:31.072779 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.072733    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bc20656-1ded-4021-bd73-d6dcd469f19d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:31.072909 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.072805    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb7gw\" (UniqueName: \"kubernetes.io/projected/8bc20656-1ded-4021-bd73-d6dcd469f19d-kube-api-access-rb7gw\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:31.072909 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.072846    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qsfm\" (UniqueName: \"kubernetes.io/projected/754f4c06-cff3-4f1d-816f-2276676045ca-kube-api-access-8qsfm\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:31.073577 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.073544    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bc20656-1ded-4021-bd73-d6dcd469f19d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:31.073942 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.073919    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bc20656-1ded-4021-bd73-d6dcd469f19d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:31.074040 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:10:31.074028    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Aug 04 18:10:31.074099 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:10:31.074082    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc20656-1ded-4021-bd73-d6dcd469f19d-proxy-tls podName:8bc20656-1ded-4021-bd73-d6dcd469f19d nodeName:}" failed. No retries permitted until 2026-08-04 18:10:31.57406461 +0000 UTC m=+634.544708944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8bc20656-1ded-4021-bd73-d6dcd469f19d-proxy-tls") pod "isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" (UID: "8bc20656-1ded-4021-bd73-d6dcd469f19d") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Aug 04 18:10:31.074166 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.074105    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/754f4c06-cff3-4f1d-816f-2276676045ca-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:31.074776 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.074743    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/754f4c06-cff3-4f1d-816f-2276676045ca-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:31.074903 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.074883    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bc20656-1ded-4021-bd73-d6dcd469f19d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:31.075082 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.075057    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/754f4c06-cff3-4f1d-816f-2276676045ca-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:31.087217 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.087185    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qsfm\" (UniqueName: \"kubernetes.io/projected/754f4c06-cff3-4f1d-816f-2276676045ca-kube-api-access-8qsfm\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:31.087403 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.087331    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb7gw\" (UniqueName: \"kubernetes.io/projected/8bc20656-1ded-4021-bd73-d6dcd469f19d-kube-api-access-rb7gw\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:31.153627 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.153599    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"]
Aug 04 18:10:31.156598 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:10:31.156557    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936d63e2_5a19_4682_ac7e_6797a64783de.slice/crio-a2b541792f7613ac8c154a9427f01ef4de436b7c0527d7fa3e8bd45ba4b02404 WatchSource:0}: Error finding container a2b541792f7613ac8c154a9427f01ef4de436b7c0527d7fa3e8bd45ba4b02404: Status 404 returned error can't find the container with id a2b541792f7613ac8c154a9427f01ef4de436b7c0527d7fa3e8bd45ba4b02404
Aug 04 18:10:31.577949 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.577918    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bc20656-1ded-4021-bd73-d6dcd469f19d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:31.578159 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.577980    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/754f4c06-cff3-4f1d-816f-2276676045ca-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:31.580483 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.580449    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bc20656-1ded-4021-bd73-d6dcd469f19d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:31.580483 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.580462    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/754f4c06-cff3-4f1d-816f-2276676045ca-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-9j6k4\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:31.722744 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.722703    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:10:31.808802 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.808754    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:10:31.871635 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.871523    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"]
Aug 04 18:10:31.874125 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:10:31.874072    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754f4c06_cff3_4f1d_816f_2276676045ca.slice/crio-ee026d9c7148ef42c06b3a8c10a488647b0f424e392747dd5d0226adaed9ffb0 WatchSource:0}: Error finding container ee026d9c7148ef42c06b3a8c10a488647b0f424e392747dd5d0226adaed9ffb0: Status 404 returned error can't find the container with id ee026d9c7148ef42c06b3a8c10a488647b0f424e392747dd5d0226adaed9ffb0
Aug 04 18:10:31.972351 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:31.972191    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"]
Aug 04 18:10:31.974501 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:10:31.974464    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc20656_1ded_4021_bd73_d6dcd469f19d.slice/crio-047d8825fc77bf3eee9994fb6dab1ff8767af4b07ce991609f2cf8dccb27d266 WatchSource:0}: Error finding container 047d8825fc77bf3eee9994fb6dab1ff8767af4b07ce991609f2cf8dccb27d266: Status 404 returned error can't find the container with id 047d8825fc77bf3eee9994fb6dab1ff8767af4b07ce991609f2cf8dccb27d266
Aug 04 18:10:32.135178 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:32.135072    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" event={"ID":"936d63e2-5a19-4682-ac7e-6797a64783de","Type":"ContainerStarted","Data":"e6d014f0ebd16957b97dff9a0c9b3c53436a5d992607f06886f0eb5989395b0a"}
Aug 04 18:10:32.135178 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:32.135162    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" event={"ID":"936d63e2-5a19-4682-ac7e-6797a64783de","Type":"ContainerStarted","Data":"a2b541792f7613ac8c154a9427f01ef4de436b7c0527d7fa3e8bd45ba4b02404"}
Aug 04 18:10:32.136895 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:32.136864    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" event={"ID":"8bc20656-1ded-4021-bd73-d6dcd469f19d","Type":"ContainerStarted","Data":"582b4d1ed6365744f53ead7b73b9f2d845a56112b477cafb6f253e9382438a5b"}
Aug 04 18:10:32.137056 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:32.136927    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" event={"ID":"8bc20656-1ded-4021-bd73-d6dcd469f19d","Type":"ContainerStarted","Data":"047d8825fc77bf3eee9994fb6dab1ff8767af4b07ce991609f2cf8dccb27d266"}
Aug 04 18:10:32.138522 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:32.138490    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" event={"ID":"754f4c06-cff3-4f1d-816f-2276676045ca","Type":"ContainerStarted","Data":"147962b3a758a2b5827c11bfb1afdfb345ddbe83d632e02ab91adf3a24bfe91b"}
Aug 04 18:10:32.138649 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:32.138549    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" event={"ID":"754f4c06-cff3-4f1d-816f-2276676045ca","Type":"ContainerStarted","Data":"ee026d9c7148ef42c06b3a8c10a488647b0f424e392747dd5d0226adaed9ffb0"}
Aug 04 18:10:35.153156 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:35.153118    2543 generic.go:356] "Generic (PLEG): container finished" podID="936d63e2-5a19-4682-ac7e-6797a64783de" containerID="e6d014f0ebd16957b97dff9a0c9b3c53436a5d992607f06886f0eb5989395b0a" exitCode=0
Aug 04 18:10:35.153559 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:35.153189    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" event={"ID":"936d63e2-5a19-4682-ac7e-6797a64783de","Type":"ContainerDied","Data":"e6d014f0ebd16957b97dff9a0c9b3c53436a5d992607f06886f0eb5989395b0a"}
Aug 04 18:10:36.163254 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:36.162179    2543 generic.go:356] "Generic (PLEG): container finished" podID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerID="582b4d1ed6365744f53ead7b73b9f2d845a56112b477cafb6f253e9382438a5b" exitCode=0
Aug 04 18:10:36.163254 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:36.162284    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" event={"ID":"8bc20656-1ded-4021-bd73-d6dcd469f19d","Type":"ContainerDied","Data":"582b4d1ed6365744f53ead7b73b9f2d845a56112b477cafb6f253e9382438a5b"}
Aug 04 18:10:36.168634 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:36.168595    2543 generic.go:356] "Generic (PLEG): container finished" podID="754f4c06-cff3-4f1d-816f-2276676045ca" containerID="147962b3a758a2b5827c11bfb1afdfb345ddbe83d632e02ab91adf3a24bfe91b" exitCode=0
Aug 04 18:10:36.168789 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:36.168723    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" event={"ID":"754f4c06-cff3-4f1d-816f-2276676045ca","Type":"ContainerDied","Data":"147962b3a758a2b5827c11bfb1afdfb345ddbe83d632e02ab91adf3a24bfe91b"}
Aug 04 18:10:57.375641 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:57.375597    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" event={"ID":"754f4c06-cff3-4f1d-816f-2276676045ca","Type":"ContainerStarted","Data":"ff2b76beb1c02ff27526d17b2fc0cb9d09ce436a5cfc74efa42adae5593ba046"}
Aug 04 18:10:57.378836 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:57.378800    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" event={"ID":"936d63e2-5a19-4682-ac7e-6797a64783de","Type":"ContainerStarted","Data":"62f96a873600e2be7d061529dca8657551c0bf424ab6d320cbcae718c4e04d2f"}
Aug 04 18:10:57.383375 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:10:57.383298    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" event={"ID":"8bc20656-1ded-4021-bd73-d6dcd469f19d","Type":"ContainerStarted","Data":"b99a4ddaf5d3b29f9e2279ce80272dd1528e829c8dc1e9a43d37666714412208"}
Aug 04 18:11:00.410440 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:00.410398    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" event={"ID":"8bc20656-1ded-4021-bd73-d6dcd469f19d","Type":"ContainerStarted","Data":"bc2cc734ce539446414be32ef51a50ba438e9f4c632b6947da2c3264d3df3df2"}
Aug 04 18:11:00.410940 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:00.410508    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:11:00.413227 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:00.413193    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" event={"ID":"754f4c06-cff3-4f1d-816f-2276676045ca","Type":"ContainerStarted","Data":"b36cdca26407f0689127527dc364af4d34a87d8966cf123467d931bd6bac0491"}
Aug 04 18:11:00.413423 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:00.413274    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:11:00.413423 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:00.413370    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:11:00.414940 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:00.414787    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:11:00.416590 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:00.416559    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" event={"ID":"936d63e2-5a19-4682-ac7e-6797a64783de","Type":"ContainerStarted","Data":"1554867b34402941fc3f2e32f2f3e428f090f1be020c774fe2a1aa4be810ce26"}
Aug 04 18:11:00.433159 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:00.433095    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" podStartSLOduration=3.298776496 podStartE2EDuration="30.433071775s" totalImagesPullingTime="23.118115045s" totalInitContainerRuntime="4.016180234s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:00.431359937 +0000 UTC m=+663.402004279" watchObservedRunningTime="2026-08-04 18:11:00.433071775 +0000 UTC m=+663.403716117"
Aug 04 18:11:00.452068 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:00.451999    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" podStartSLOduration=3.860000989 podStartE2EDuration="30.451979368s" totalImagesPullingTime="23.084623529s" totalInitContainerRuntime="3.50735485s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:00.450278166 +0000 UTC m=+663.420922545" watchObservedRunningTime="2026-08-04 18:11:00.451979368 +0000 UTC m=+663.422623710"
Aug 04 18:11:01.421581 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:01.421533    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:11:01.422036 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:01.421676    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:11:01.422870 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:01.422844    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 04 18:11:02.426876 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:02.426826    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 04 18:11:03.432690 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:03.432647    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" event={"ID":"936d63e2-5a19-4682-ac7e-6797a64783de","Type":"ContainerStarted","Data":"f5b5bf85972d96dd85bb667c1561b17975f549de37b23ce40b5d8eba427feeb1"}
Aug 04 18:11:03.433147 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:03.432843    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:11:03.461065 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:03.460996    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podStartSLOduration=2.517498303 podStartE2EDuration="33.460968958s" totalImagesPullingTime="27.218724103s" totalInitContainerRuntime="3.724746552s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:03.458790012 +0000 UTC m=+666.429434364" watchObservedRunningTime="2026-08-04 18:11:03.460968958 +0000 UTC m=+666.431613299"
Aug 04 18:11:04.437278 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:04.437238    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:11:04.437696 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:04.437386    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:11:04.439116 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:04.439079    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:11:04.439945 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:04.439916    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:11:04.442906 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:04.442883    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:11:05.441909 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:05.441858    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:11:05.442333 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:05.442192    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:11:06.426088 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:06.426059    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:11:06.426599 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:06.426570    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:11:06.446111 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:06.446073    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:11:06.446549 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:06.446447    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:11:07.431832 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:07.431799    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:11:07.432479 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:07.432446    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 04 18:11:16.426620 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:16.426574    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:11:16.446597 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:16.446542    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:11:16.446922 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:16.446900    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:11:17.432630 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:17.432593    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 04 18:11:26.426545 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:26.426502    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:11:26.446460 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:26.446424    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:11:26.446886 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:26.446860    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:11:27.432462 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:27.432424    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 04 18:11:36.427163 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:36.427123    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:11:36.445970 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:36.445932    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:11:36.446374 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:36.446353    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:11:37.432894 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:37.432857    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 04 18:11:46.426527 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:46.426488    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:11:46.446061 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:46.446008    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:11:46.446531 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:46.446503    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:11:47.433111 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:47.433062    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 04 18:11:56.426774 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:56.426693    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:11:56.446414 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:56.446381    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:11:56.446791 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:56.446771    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:11:57.433326 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:11:57.433270    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 04 18:12:06.427023 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:06.426982    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:12:06.446069 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:06.446034    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:12:06.446646 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:06.446620    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:12:06.569853 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:06.569814    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:12:07.432813 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:07.432773    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 04 18:12:15.574060 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:15.574033    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:12:16.446514 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:16.446484    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:12:16.446678 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:16.446575    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:12:16.571049 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:16.571023    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:12:20.546260 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:20.546226    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"]
Aug 04 18:12:20.546858 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:20.546513    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kserve-container" containerID="cri-o://b99a4ddaf5d3b29f9e2279ce80272dd1528e829c8dc1e9a43d37666714412208" gracePeriod=30
Aug 04 18:12:20.546858 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:20.546559    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kube-rbac-proxy" containerID="cri-o://bc2cc734ce539446414be32ef51a50ba438e9f4c632b6947da2c3264d3df3df2" gracePeriod=30
Aug 04 18:12:20.762263 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:20.762223    2543 generic.go:356] "Generic (PLEG): container finished" podID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerID="bc2cc734ce539446414be32ef51a50ba438e9f4c632b6947da2c3264d3df3df2" exitCode=2
Aug 04 18:12:20.762453 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:20.762270    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" event={"ID":"8bc20656-1ded-4021-bd73-d6dcd469f19d","Type":"ContainerDied","Data":"bc2cc734ce539446414be32ef51a50ba438e9f4c632b6947da2c3264d3df3df2"}
Aug 04 18:12:22.427952 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:22.427911    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Aug 04 18:12:25.484494 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.484471    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:12:25.515337 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.515251    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bc20656-1ded-4021-bd73-d6dcd469f19d-cabundle-cert\") pod \"8bc20656-1ded-4021-bd73-d6dcd469f19d\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") "
Aug 04 18:12:25.515497 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.515339    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bc20656-1ded-4021-bd73-d6dcd469f19d-kserve-provision-location\") pod \"8bc20656-1ded-4021-bd73-d6dcd469f19d\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") "
Aug 04 18:12:25.515497 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.515361    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bc20656-1ded-4021-bd73-d6dcd469f19d-kube-rbac-proxy-sar-config\") pod \"8bc20656-1ded-4021-bd73-d6dcd469f19d\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") "
Aug 04 18:12:25.515497 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.515379    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bc20656-1ded-4021-bd73-d6dcd469f19d-proxy-tls\") pod \"8bc20656-1ded-4021-bd73-d6dcd469f19d\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") "
Aug 04 18:12:25.515497 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.515480    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb7gw\" (UniqueName: \"kubernetes.io/projected/8bc20656-1ded-4021-bd73-d6dcd469f19d-kube-api-access-rb7gw\") pod \"8bc20656-1ded-4021-bd73-d6dcd469f19d\" (UID: \"8bc20656-1ded-4021-bd73-d6dcd469f19d\") "
Aug 04 18:12:25.515716 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.515686    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc20656-1ded-4021-bd73-d6dcd469f19d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8bc20656-1ded-4021-bd73-d6dcd469f19d" (UID: "8bc20656-1ded-4021-bd73-d6dcd469f19d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:12:25.515777 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.515711    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc20656-1ded-4021-bd73-d6dcd469f19d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8bc20656-1ded-4021-bd73-d6dcd469f19d" (UID: "8bc20656-1ded-4021-bd73-d6dcd469f19d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:12:25.515777 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.515722    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc20656-1ded-4021-bd73-d6dcd469f19d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8bc20656-1ded-4021-bd73-d6dcd469f19d" (UID: "8bc20656-1ded-4021-bd73-d6dcd469f19d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:12:25.515870 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.515851    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bc20656-1ded-4021-bd73-d6dcd469f19d-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:12:25.515905 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.515876    2543 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bc20656-1ded-4021-bd73-d6dcd469f19d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:12:25.515905 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.515890    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bc20656-1ded-4021-bd73-d6dcd469f19d-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:12:25.517643 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.517619    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc20656-1ded-4021-bd73-d6dcd469f19d-kube-api-access-rb7gw" (OuterVolumeSpecName: "kube-api-access-rb7gw") pod "8bc20656-1ded-4021-bd73-d6dcd469f19d" (UID: "8bc20656-1ded-4021-bd73-d6dcd469f19d"). InnerVolumeSpecName "kube-api-access-rb7gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:12:25.517747 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.517680    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc20656-1ded-4021-bd73-d6dcd469f19d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8bc20656-1ded-4021-bd73-d6dcd469f19d" (UID: "8bc20656-1ded-4021-bd73-d6dcd469f19d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:12:25.617363 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.617331    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bc20656-1ded-4021-bd73-d6dcd469f19d-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:12:25.617509 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.617367    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb7gw\" (UniqueName: \"kubernetes.io/projected/8bc20656-1ded-4021-bd73-d6dcd469f19d-kube-api-access-rb7gw\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:12:25.779903 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.779816    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"]
Aug 04 18:12:25.780157 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.780104    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container" containerID="cri-o://62f96a873600e2be7d061529dca8657551c0bf424ab6d320cbcae718c4e04d2f" gracePeriod=30
Aug 04 18:12:25.780263 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.780228    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kube-rbac-proxy" containerID="cri-o://1554867b34402941fc3f2e32f2f3e428f090f1be020c774fe2a1aa4be810ce26" gracePeriod=30
Aug 04 18:12:25.780389 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.780147    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent" containerID="cri-o://f5b5bf85972d96dd85bb667c1561b17975f549de37b23ce40b5d8eba427feeb1" gracePeriod=30
Aug 04 18:12:25.787875 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.787851    2543 generic.go:356] "Generic (PLEG): container finished" podID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerID="b99a4ddaf5d3b29f9e2279ce80272dd1528e829c8dc1e9a43d37666714412208" exitCode=0
Aug 04 18:12:25.787983 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.787893    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" event={"ID":"8bc20656-1ded-4021-bd73-d6dcd469f19d","Type":"ContainerDied","Data":"b99a4ddaf5d3b29f9e2279ce80272dd1528e829c8dc1e9a43d37666714412208"}
Aug 04 18:12:25.787983 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.787915    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"
Aug 04 18:12:25.787983 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.787951    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm" event={"ID":"8bc20656-1ded-4021-bd73-d6dcd469f19d","Type":"ContainerDied","Data":"047d8825fc77bf3eee9994fb6dab1ff8767af4b07ce991609f2cf8dccb27d266"}
Aug 04 18:12:25.788152 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.787989    2543 scope.go:117] "RemoveContainer" containerID="bc2cc734ce539446414be32ef51a50ba438e9f4c632b6947da2c3264d3df3df2"
Aug 04 18:12:25.819131 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.819102    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"]
Aug 04 18:12:25.831439 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.831414    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-jfmwm"]
Aug 04 18:12:25.890287 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.890256    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"]
Aug 04 18:12:25.890640 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.890625    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kserve-container"
Aug 04 18:12:25.890640 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.890640    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kserve-container"
Aug 04 18:12:25.890804 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.890649    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kube-rbac-proxy"
Aug 04 18:12:25.890804 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.890655    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kube-rbac-proxy"
Aug 04 18:12:25.890804 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.890666    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="storage-initializer"
Aug 04 18:12:25.890804 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.890671    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="storage-initializer"
Aug 04 18:12:25.890804 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.890729    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kube-rbac-proxy"
Aug 04 18:12:25.890804 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.890736    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" containerName="kserve-container"
Aug 04 18:12:25.891789 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.891759    2543 scope.go:117] "RemoveContainer" containerID="b99a4ddaf5d3b29f9e2279ce80272dd1528e829c8dc1e9a43d37666714412208"
Aug 04 18:12:25.895148 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.895124    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:25.897770 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.897749    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-c55f6-predictor-serving-cert\""
Aug 04 18:12:25.897869 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.897770    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-c55f6-kube-rbac-proxy-sar-config\""
Aug 04 18:12:25.900570 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.900556    2543 scope.go:117] "RemoveContainer" containerID="582b4d1ed6365744f53ead7b73b9f2d845a56112b477cafb6f253e9382438a5b"
Aug 04 18:12:25.908217 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.908193    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"]
Aug 04 18:12:25.910847 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.910694    2543 scope.go:117] "RemoveContainer" containerID="bc2cc734ce539446414be32ef51a50ba438e9f4c632b6947da2c3264d3df3df2"
Aug 04 18:12:25.911024 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:12:25.911005    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2cc734ce539446414be32ef51a50ba438e9f4c632b6947da2c3264d3df3df2\": container with ID starting with bc2cc734ce539446414be32ef51a50ba438e9f4c632b6947da2c3264d3df3df2 not found: ID does not exist" containerID="bc2cc734ce539446414be32ef51a50ba438e9f4c632b6947da2c3264d3df3df2"
Aug 04 18:12:25.911081 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.911034    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2cc734ce539446414be32ef51a50ba438e9f4c632b6947da2c3264d3df3df2"} err="failed to get container status \"bc2cc734ce539446414be32ef51a50ba438e9f4c632b6947da2c3264d3df3df2\": rpc error: code = NotFound desc = could not find container \"bc2cc734ce539446414be32ef51a50ba438e9f4c632b6947da2c3264d3df3df2\": container with ID starting with bc2cc734ce539446414be32ef51a50ba438e9f4c632b6947da2c3264d3df3df2 not found: ID does not exist"
Aug 04 18:12:25.911081 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.911055    2543 scope.go:117] "RemoveContainer" containerID="b99a4ddaf5d3b29f9e2279ce80272dd1528e829c8dc1e9a43d37666714412208"
Aug 04 18:12:25.911775 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:12:25.911750    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b99a4ddaf5d3b29f9e2279ce80272dd1528e829c8dc1e9a43d37666714412208\": container with ID starting with b99a4ddaf5d3b29f9e2279ce80272dd1528e829c8dc1e9a43d37666714412208 not found: ID does not exist" containerID="b99a4ddaf5d3b29f9e2279ce80272dd1528e829c8dc1e9a43d37666714412208"
Aug 04 18:12:25.911902 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.911783    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b99a4ddaf5d3b29f9e2279ce80272dd1528e829c8dc1e9a43d37666714412208"} err="failed to get container status \"b99a4ddaf5d3b29f9e2279ce80272dd1528e829c8dc1e9a43d37666714412208\": rpc error: code = NotFound desc = could not find container \"b99a4ddaf5d3b29f9e2279ce80272dd1528e829c8dc1e9a43d37666714412208\": container with ID starting with b99a4ddaf5d3b29f9e2279ce80272dd1528e829c8dc1e9a43d37666714412208 not found: ID does not exist"
Aug 04 18:12:25.911902 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.911806    2543 scope.go:117] "RemoveContainer" containerID="582b4d1ed6365744f53ead7b73b9f2d845a56112b477cafb6f253e9382438a5b"
Aug 04 18:12:25.912144 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:12:25.912117    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582b4d1ed6365744f53ead7b73b9f2d845a56112b477cafb6f253e9382438a5b\": container with ID starting with 582b4d1ed6365744f53ead7b73b9f2d845a56112b477cafb6f253e9382438a5b not found: ID does not exist" containerID="582b4d1ed6365744f53ead7b73b9f2d845a56112b477cafb6f253e9382438a5b"
Aug 04 18:12:25.912242 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.912154    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582b4d1ed6365744f53ead7b73b9f2d845a56112b477cafb6f253e9382438a5b"} err="failed to get container status \"582b4d1ed6365744f53ead7b73b9f2d845a56112b477cafb6f253e9382438a5b\": rpc error: code = NotFound desc = could not find container \"582b4d1ed6365744f53ead7b73b9f2d845a56112b477cafb6f253e9382438a5b\": container with ID starting with 582b4d1ed6365744f53ead7b73b9f2d845a56112b477cafb6f253e9382438a5b not found: ID does not exist"
Aug 04 18:12:25.920608 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.920581    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b09ebb2d-7f40-4ebd-a850-c58d32199f96-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:25.920711 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.920631    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:25.920711 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.920674    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:25.920814 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.920722    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq2wc\" (UniqueName: \"kubernetes.io/projected/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kube-api-access-jq2wc\") pod \"isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:25.920814 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.920763    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b09ebb2d-7f40-4ebd-a850-c58d32199f96-proxy-tls\") pod \"isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:25.997563 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:25.997532    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"]
Aug 04 18:12:26.003683 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.003661    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.007013 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.006991    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-c55f6-predictor-serving-cert\""
Aug 04 18:12:26.007104 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.007075    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-c55f6-kube-rbac-proxy-sar-config\""
Aug 04 18:12:26.012740 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.012719    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"]
Aug 04 18:12:26.021582 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.021561    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52069c82-27a0-4aeb-9be5-63288ef62e3b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.021655 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.021598    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b09ebb2d-7f40-4ebd-a850-c58d32199f96-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:26.021655 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.021621    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:26.021754 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.021672    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:26.021754 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.021701    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq2wc\" (UniqueName: \"kubernetes.io/projected/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kube-api-access-jq2wc\") pod \"isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:26.021754 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.021750    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b09ebb2d-7f40-4ebd-a850-c58d32199f96-proxy-tls\") pod \"isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:26.021902 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.021797    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52069c82-27a0-4aeb-9be5-63288ef62e3b-proxy-tls\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.021902 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.021845    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52069c82-27a0-4aeb-9be5-63288ef62e3b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.021902 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.021882    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52069c82-27a0-4aeb-9be5-63288ef62e3b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.022047 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.021922    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj2n2\" (UniqueName: \"kubernetes.io/projected/52069c82-27a0-4aeb-9be5-63288ef62e3b-kube-api-access-kj2n2\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.022114 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.022058    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:26.022355 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.022333    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b09ebb2d-7f40-4ebd-a850-c58d32199f96-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:26.022355 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.022347    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:26.024717 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.024693    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b09ebb2d-7f40-4ebd-a850-c58d32199f96-proxy-tls\") pod \"isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:26.030497 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.030445    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq2wc\" (UniqueName: \"kubernetes.io/projected/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kube-api-access-jq2wc\") pod \"isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:26.122747 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.122714    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52069c82-27a0-4aeb-9be5-63288ef62e3b-proxy-tls\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.122916 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.122756    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52069c82-27a0-4aeb-9be5-63288ef62e3b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.122916 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.122781    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52069c82-27a0-4aeb-9be5-63288ef62e3b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.122916 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.122822    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj2n2\" (UniqueName: \"kubernetes.io/projected/52069c82-27a0-4aeb-9be5-63288ef62e3b-kube-api-access-kj2n2\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.122916 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.122863    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52069c82-27a0-4aeb-9be5-63288ef62e3b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.122916 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:12:26.122881    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-serving-cert: secret "isvc-xgboost-graph-raw-c55f6-predictor-serving-cert" not found
Aug 04 18:12:26.123416 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:12:26.122961    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52069c82-27a0-4aeb-9be5-63288ef62e3b-proxy-tls podName:52069c82-27a0-4aeb-9be5-63288ef62e3b nodeName:}" failed. No retries permitted until 2026-08-04 18:12:26.622940244 +0000 UTC m=+749.593584579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/52069c82-27a0-4aeb-9be5-63288ef62e3b-proxy-tls") pod "isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" (UID: "52069c82-27a0-4aeb-9be5-63288ef62e3b") : secret "isvc-xgboost-graph-raw-c55f6-predictor-serving-cert" not found
Aug 04 18:12:26.123416 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.123207    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52069c82-27a0-4aeb-9be5-63288ef62e3b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.123543 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.123524    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52069c82-27a0-4aeb-9be5-63288ef62e3b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.123579 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.123524    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52069c82-27a0-4aeb-9be5-63288ef62e3b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.132113 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.132095    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj2n2\" (UniqueName: \"kubernetes.io/projected/52069c82-27a0-4aeb-9be5-63288ef62e3b-kube-api-access-kj2n2\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.206648 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.206613    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:26.345413 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.345372    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"]
Aug 04 18:12:26.347682 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:12:26.347647    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09ebb2d_7f40_4ebd_a850_c58d32199f96.slice/crio-db7649dcffd04982ccbd2e5e9e2fab4f9be717b800d6311e0a1e23c60191d3aa WatchSource:0}: Error finding container db7649dcffd04982ccbd2e5e9e2fab4f9be717b800d6311e0a1e23c60191d3aa: Status 404 returned error can't find the container with id db7649dcffd04982ccbd2e5e9e2fab4f9be717b800d6311e0a1e23c60191d3aa
Aug 04 18:12:26.446869 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.446820    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:12:26.447157 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.447124    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:12:26.626623 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.626592    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52069c82-27a0-4aeb-9be5-63288ef62e3b-proxy-tls\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.628920 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.628895    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52069c82-27a0-4aeb-9be5-63288ef62e3b-proxy-tls\") pod \"isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:26.795417 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.795385    2543 generic.go:356] "Generic (PLEG): container finished" podID="936d63e2-5a19-4682-ac7e-6797a64783de" containerID="1554867b34402941fc3f2e32f2f3e428f090f1be020c774fe2a1aa4be810ce26" exitCode=2
Aug 04 18:12:26.795577 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.795468    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" event={"ID":"936d63e2-5a19-4682-ac7e-6797a64783de","Type":"ContainerDied","Data":"1554867b34402941fc3f2e32f2f3e428f090f1be020c774fe2a1aa4be810ce26"}
Aug 04 18:12:26.796940 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.796869    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" event={"ID":"b09ebb2d-7f40-4ebd-a850-c58d32199f96","Type":"ContainerStarted","Data":"4b0902decba1ef099affb424231792f379c751ba9efe4b66602e894c235c8ec9"}
Aug 04 18:12:26.796940 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.796929    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" event={"ID":"b09ebb2d-7f40-4ebd-a850-c58d32199f96","Type":"ContainerStarted","Data":"db7649dcffd04982ccbd2e5e9e2fab4f9be717b800d6311e0a1e23c60191d3aa"}
Aug 04 18:12:26.914238 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:26.914157    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:27.048482 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:27.048458    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"]
Aug 04 18:12:27.050506 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:12:27.050483    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52069c82_27a0_4aeb_9be5_63288ef62e3b.slice/crio-8781b1e1c80d0e94c26a25375e66eebc31047ad1c22180f722198fe709a75984 WatchSource:0}: Error finding container 8781b1e1c80d0e94c26a25375e66eebc31047ad1c22180f722198fe709a75984: Status 404 returned error can't find the container with id 8781b1e1c80d0e94c26a25375e66eebc31047ad1c22180f722198fe709a75984
Aug 04 18:12:27.574871 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:27.574839    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bc20656-1ded-4021-bd73-d6dcd469f19d" path="/var/lib/kubelet/pods/8bc20656-1ded-4021-bd73-d6dcd469f19d/volumes"
Aug 04 18:12:27.803826 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:27.803792    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" event={"ID":"52069c82-27a0-4aeb-9be5-63288ef62e3b","Type":"ContainerStarted","Data":"fdcf227031f979f26aac37b06021959782ecf3f5feb5741e30c21ed6088cd466"}
Aug 04 18:12:27.804264 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:27.803848    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" event={"ID":"52069c82-27a0-4aeb-9be5-63288ef62e3b","Type":"ContainerStarted","Data":"8781b1e1c80d0e94c26a25375e66eebc31047ad1c22180f722198fe709a75984"}
Aug 04 18:12:29.437787 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:29.437746    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Aug 04 18:12:30.819582 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:30.819508    2543 generic.go:356] "Generic (PLEG): container finished" podID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerID="fdcf227031f979f26aac37b06021959782ecf3f5feb5741e30c21ed6088cd466" exitCode=0
Aug 04 18:12:30.819980 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:30.819583    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" event={"ID":"52069c82-27a0-4aeb-9be5-63288ef62e3b","Type":"ContainerDied","Data":"fdcf227031f979f26aac37b06021959782ecf3f5feb5741e30c21ed6088cd466"}
Aug 04 18:12:30.822223 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:30.822204    2543 generic.go:356] "Generic (PLEG): container finished" podID="936d63e2-5a19-4682-ac7e-6797a64783de" containerID="62f96a873600e2be7d061529dca8657551c0bf424ab6d320cbcae718c4e04d2f" exitCode=0
Aug 04 18:12:30.822296 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:30.822272    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" event={"ID":"936d63e2-5a19-4682-ac7e-6797a64783de","Type":"ContainerDied","Data":"62f96a873600e2be7d061529dca8657551c0bf424ab6d320cbcae718c4e04d2f"}
Aug 04 18:12:30.823549 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:30.823531    2543 generic.go:356] "Generic (PLEG): container finished" podID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerID="4b0902decba1ef099affb424231792f379c751ba9efe4b66602e894c235c8ec9" exitCode=0
Aug 04 18:12:30.823618 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:30.823572    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" event={"ID":"b09ebb2d-7f40-4ebd-a850-c58d32199f96","Type":"ContainerDied","Data":"4b0902decba1ef099affb424231792f379c751ba9efe4b66602e894c235c8ec9"}
Aug 04 18:12:31.834651 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:31.834604    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" event={"ID":"b09ebb2d-7f40-4ebd-a850-c58d32199f96","Type":"ContainerStarted","Data":"576190922cca686f452146a426251f1fd16df69ed5116435d4964c61d17ab579"}
Aug 04 18:12:31.835208 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:31.834712    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" event={"ID":"b09ebb2d-7f40-4ebd-a850-c58d32199f96","Type":"ContainerStarted","Data":"8ae52486e75d3c01435322721a339998138f2bcb882dc3fbe3138b3e27398be7"}
Aug 04 18:12:31.835208 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:31.835008    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:32.842223 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:32.842194    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:32.843700 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:32.843667    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:12:33.847624 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:33.847582    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:12:34.438049 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:34.438001    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Aug 04 18:12:36.446553 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:36.446511    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:12:36.447043 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:36.446833    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:12:38.853483 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:38.853449    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:12:38.854071 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:38.854044    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:12:38.875908 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:38.875857    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" podStartSLOduration=10.301472146 podStartE2EDuration="13.875842347s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.574370201s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:31.855267127 +0000 UTC m=+754.825911470" watchObservedRunningTime="2026-08-04 18:12:38.875842347 +0000 UTC m=+761.846486688"
Aug 04 18:12:39.438501 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:39.438456    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Aug 04 18:12:39.438697 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:39.438617    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:12:44.438034 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:44.437987    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Aug 04 18:12:46.446018 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:46.445966    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:12:46.446517 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:46.446168    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:12:46.446517 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:46.446263    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:12:46.446517 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:46.446409    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:12:48.854491 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:48.854454    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:12:48.930528 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:48.930499    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" event={"ID":"52069c82-27a0-4aeb-9be5-63288ef62e3b","Type":"ContainerStarted","Data":"ddcbf66b2a53c5a409dde701afe96387930cf527a8f62566c46d5f0ccd650e02"}
Aug 04 18:12:48.930682 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:48.930556    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" event={"ID":"52069c82-27a0-4aeb-9be5-63288ef62e3b","Type":"ContainerStarted","Data":"f4adae7689cffe2d0ee74ec70a1d610bb4e3f70565f744c6ed9b5087c1bac97b"}
Aug 04 18:12:48.930751 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:48.930736    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:48.952434 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:48.952394    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" podStartSLOduration=3.235383102 podStartE2EDuration="23.9523805s" totalImagesPullingTime="17.258082068s" totalInitContainerRuntime="3.45891533s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:12:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:48.95204338 +0000 UTC m=+771.922687720" watchObservedRunningTime="2026-08-04 18:12:48.9523805 +0000 UTC m=+771.923024841"
Aug 04 18:12:49.438045 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:49.438006    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Aug 04 18:12:49.935371 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:49.935334    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:49.936554 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:49.936523    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:12:50.940020 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:50.939963    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:12:54.437833 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:54.437791    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Aug 04 18:12:55.945758 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:55.945731    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:12:55.946220 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:55.946193    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:12:55.966730 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:55.966596    2543 generic.go:356] "Generic (PLEG): container finished" podID="936d63e2-5a19-4682-ac7e-6797a64783de" containerID="f5b5bf85972d96dd85bb667c1561b17975f549de37b23ce40b5d8eba427feeb1" exitCode=0
Aug 04 18:12:55.966730 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:55.966704    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" event={"ID":"936d63e2-5a19-4682-ac7e-6797a64783de","Type":"ContainerDied","Data":"f5b5bf85972d96dd85bb667c1561b17975f549de37b23ce40b5d8eba427feeb1"}
Aug 04 18:12:56.492721 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.492694    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:12:56.602071 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.602048    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/936d63e2-5a19-4682-ac7e-6797a64783de-cabundle-cert\") pod \"936d63e2-5a19-4682-ac7e-6797a64783de\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") "
Aug 04 18:12:56.602208 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.602097    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/936d63e2-5a19-4682-ac7e-6797a64783de-kube-rbac-proxy-sar-config\") pod \"936d63e2-5a19-4682-ac7e-6797a64783de\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") "
Aug 04 18:12:56.602208 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.602130    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/936d63e2-5a19-4682-ac7e-6797a64783de-kserve-provision-location\") pod \"936d63e2-5a19-4682-ac7e-6797a64783de\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") "
Aug 04 18:12:56.602208 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.602179    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/936d63e2-5a19-4682-ac7e-6797a64783de-proxy-tls\") pod \"936d63e2-5a19-4682-ac7e-6797a64783de\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") "
Aug 04 18:12:56.602208 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.602197    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz68h\" (UniqueName: \"kubernetes.io/projected/936d63e2-5a19-4682-ac7e-6797a64783de-kube-api-access-tz68h\") pod \"936d63e2-5a19-4682-ac7e-6797a64783de\" (UID: \"936d63e2-5a19-4682-ac7e-6797a64783de\") "
Aug 04 18:12:56.602511 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.602483    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936d63e2-5a19-4682-ac7e-6797a64783de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "936d63e2-5a19-4682-ac7e-6797a64783de" (UID: "936d63e2-5a19-4682-ac7e-6797a64783de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:12:56.602580 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.602525    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936d63e2-5a19-4682-ac7e-6797a64783de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "936d63e2-5a19-4682-ac7e-6797a64783de" (UID: "936d63e2-5a19-4682-ac7e-6797a64783de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:12:56.602580 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.602553    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936d63e2-5a19-4682-ac7e-6797a64783de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "936d63e2-5a19-4682-ac7e-6797a64783de" (UID: "936d63e2-5a19-4682-ac7e-6797a64783de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:12:56.604297 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.604274    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936d63e2-5a19-4682-ac7e-6797a64783de-kube-api-access-tz68h" (OuterVolumeSpecName: "kube-api-access-tz68h") pod "936d63e2-5a19-4682-ac7e-6797a64783de" (UID: "936d63e2-5a19-4682-ac7e-6797a64783de"). InnerVolumeSpecName "kube-api-access-tz68h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:12:56.604381 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.604291    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/936d63e2-5a19-4682-ac7e-6797a64783de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "936d63e2-5a19-4682-ac7e-6797a64783de" (UID: "936d63e2-5a19-4682-ac7e-6797a64783de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:12:56.703431 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.703404    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/936d63e2-5a19-4682-ac7e-6797a64783de-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:12:56.703431 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.703426    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz68h\" (UniqueName: \"kubernetes.io/projected/936d63e2-5a19-4682-ac7e-6797a64783de-kube-api-access-tz68h\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:12:56.703431 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.703437    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/936d63e2-5a19-4682-ac7e-6797a64783de-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:12:56.703602 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.703445    2543 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/936d63e2-5a19-4682-ac7e-6797a64783de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:12:56.703602 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.703453    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/936d63e2-5a19-4682-ac7e-6797a64783de-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:12:56.973428 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.973339    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" event={"ID":"936d63e2-5a19-4682-ac7e-6797a64783de","Type":"ContainerDied","Data":"a2b541792f7613ac8c154a9427f01ef4de436b7c0527d7fa3e8bd45ba4b02404"}
Aug 04 18:12:56.973428 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.973383    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"
Aug 04 18:12:56.973428 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.973420    2543 scope.go:117] "RemoveContainer" containerID="f5b5bf85972d96dd85bb667c1561b17975f549de37b23ce40b5d8eba427feeb1"
Aug 04 18:12:56.981610 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.981591    2543 scope.go:117] "RemoveContainer" containerID="1554867b34402941fc3f2e32f2f3e428f090f1be020c774fe2a1aa4be810ce26"
Aug 04 18:12:56.988820 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.988802    2543 scope.go:117] "RemoveContainer" containerID="62f96a873600e2be7d061529dca8657551c0bf424ab6d320cbcae718c4e04d2f"
Aug 04 18:12:56.996493 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.996477    2543 scope.go:117] "RemoveContainer" containerID="e6d014f0ebd16957b97dff9a0c9b3c53436a5d992607f06886f0eb5989395b0a"
Aug 04 18:12:56.998715 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:56.998693    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"]
Aug 04 18:12:57.002514 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:57.002495    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h"]
Aug 04 18:12:57.446977 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:57.446938    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: i/o timeout"
Aug 04 18:12:57.447161 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:57.447013    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5a068-predictor-57765876c5-dlf8h" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent" probeResult="failure" output="Get \"http://10.132.0.25:9081/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Aug 04 18:12:57.574728 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:57.574699    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" path="/var/lib/kubelet/pods/936d63e2-5a19-4682-ac7e-6797a64783de/volumes"
Aug 04 18:12:58.854023 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:12:58.853980    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:13:05.946598 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:13:05.946563    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:13:08.854226 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:13:08.854175    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:13:15.946782 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:13:15.946740    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:13:18.854126 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:13:18.854076    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:13:25.946336 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:13:25.946232    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:13:28.854464 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:13:28.854423    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:13:35.946982 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:13:35.946943    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:13:38.854195 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:13:38.854150    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:13:45.947506 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:13:45.947477    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:13:48.854486 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:13:48.854459    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:14:00.714502 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:00.714473    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"]
Aug 04 18:14:00.714997 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:00.714703    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kserve-container" containerID="cri-o://ff2b76beb1c02ff27526d17b2fc0cb9d09ce436a5cfc74efa42adae5593ba046" gracePeriod=30
Aug 04 18:14:00.714997 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:00.714747    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kube-rbac-proxy" containerID="cri-o://b36cdca26407f0689127527dc364af4d34a87d8966cf123467d931bd6bac0491" gracePeriod=30
Aug 04 18:14:01.239917 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:01.239882    2543 generic.go:356] "Generic (PLEG): container finished" podID="754f4c06-cff3-4f1d-816f-2276676045ca" containerID="b36cdca26407f0689127527dc364af4d34a87d8966cf123467d931bd6bac0491" exitCode=2
Aug 04 18:14:01.240094 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:01.239959    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" event={"ID":"754f4c06-cff3-4f1d-816f-2276676045ca","Type":"ContainerDied","Data":"b36cdca26407f0689127527dc364af4d34a87d8966cf123467d931bd6bac0491"}
Aug 04 18:14:01.422582 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:01.422546    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Aug 04 18:14:04.655875 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.655844    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:14:04.743047 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.742989    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/754f4c06-cff3-4f1d-816f-2276676045ca-kube-rbac-proxy-sar-config\") pod \"754f4c06-cff3-4f1d-816f-2276676045ca\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") "
Aug 04 18:14:04.743047 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.743022    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qsfm\" (UniqueName: \"kubernetes.io/projected/754f4c06-cff3-4f1d-816f-2276676045ca-kube-api-access-8qsfm\") pod \"754f4c06-cff3-4f1d-816f-2276676045ca\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") "
Aug 04 18:14:04.743192 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.743082    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/754f4c06-cff3-4f1d-816f-2276676045ca-proxy-tls\") pod \"754f4c06-cff3-4f1d-816f-2276676045ca\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") "
Aug 04 18:14:04.743192 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.743120    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/754f4c06-cff3-4f1d-816f-2276676045ca-kserve-provision-location\") pod \"754f4c06-cff3-4f1d-816f-2276676045ca\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") "
Aug 04 18:14:04.743272 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.743244    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/754f4c06-cff3-4f1d-816f-2276676045ca-cabundle-cert\") pod \"754f4c06-cff3-4f1d-816f-2276676045ca\" (UID: \"754f4c06-cff3-4f1d-816f-2276676045ca\") "
Aug 04 18:14:04.743414 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.743389    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754f4c06-cff3-4f1d-816f-2276676045ca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "754f4c06-cff3-4f1d-816f-2276676045ca" (UID: "754f4c06-cff3-4f1d-816f-2276676045ca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:14:04.743496 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.743444    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754f4c06-cff3-4f1d-816f-2276676045ca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "754f4c06-cff3-4f1d-816f-2276676045ca" (UID: "754f4c06-cff3-4f1d-816f-2276676045ca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:04.743555 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.743522    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/754f4c06-cff3-4f1d-816f-2276676045ca-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:14:04.743555 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.743541    2543 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/754f4c06-cff3-4f1d-816f-2276676045ca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:14:04.743669 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.743642    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754f4c06-cff3-4f1d-816f-2276676045ca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "754f4c06-cff3-4f1d-816f-2276676045ca" (UID: "754f4c06-cff3-4f1d-816f-2276676045ca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:14:04.745189 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.745167    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754f4c06-cff3-4f1d-816f-2276676045ca-kube-api-access-8qsfm" (OuterVolumeSpecName: "kube-api-access-8qsfm") pod "754f4c06-cff3-4f1d-816f-2276676045ca" (UID: "754f4c06-cff3-4f1d-816f-2276676045ca"). InnerVolumeSpecName "kube-api-access-8qsfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:14:04.745247 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.745178    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/754f4c06-cff3-4f1d-816f-2276676045ca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "754f4c06-cff3-4f1d-816f-2276676045ca" (UID: "754f4c06-cff3-4f1d-816f-2276676045ca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:14:04.844625 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.844603    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/754f4c06-cff3-4f1d-816f-2276676045ca-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:14:04.844625 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.844624    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/754f4c06-cff3-4f1d-816f-2276676045ca-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:14:04.844759 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:04.844632    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qsfm\" (UniqueName: \"kubernetes.io/projected/754f4c06-cff3-4f1d-816f-2276676045ca-kube-api-access-8qsfm\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:14:05.258382 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.258349    2543 generic.go:356] "Generic (PLEG): container finished" podID="754f4c06-cff3-4f1d-816f-2276676045ca" containerID="ff2b76beb1c02ff27526d17b2fc0cb9d09ce436a5cfc74efa42adae5593ba046" exitCode=0
Aug 04 18:14:05.258551 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.258410    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"
Aug 04 18:14:05.258551 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.258436    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" event={"ID":"754f4c06-cff3-4f1d-816f-2276676045ca","Type":"ContainerDied","Data":"ff2b76beb1c02ff27526d17b2fc0cb9d09ce436a5cfc74efa42adae5593ba046"}
Aug 04 18:14:05.258551 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.258495    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4" event={"ID":"754f4c06-cff3-4f1d-816f-2276676045ca","Type":"ContainerDied","Data":"ee026d9c7148ef42c06b3a8c10a488647b0f424e392747dd5d0226adaed9ffb0"}
Aug 04 18:14:05.258551 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.258522    2543 scope.go:117] "RemoveContainer" containerID="b36cdca26407f0689127527dc364af4d34a87d8966cf123467d931bd6bac0491"
Aug 04 18:14:05.267542 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.267525    2543 scope.go:117] "RemoveContainer" containerID="ff2b76beb1c02ff27526d17b2fc0cb9d09ce436a5cfc74efa42adae5593ba046"
Aug 04 18:14:05.274556 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.274541    2543 scope.go:117] "RemoveContainer" containerID="147962b3a758a2b5827c11bfb1afdfb345ddbe83d632e02ab91adf3a24bfe91b"
Aug 04 18:14:05.281677 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.281659    2543 scope.go:117] "RemoveContainer" containerID="b36cdca26407f0689127527dc364af4d34a87d8966cf123467d931bd6bac0491"
Aug 04 18:14:05.281939 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:14:05.281924    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36cdca26407f0689127527dc364af4d34a87d8966cf123467d931bd6bac0491\": container with ID starting with b36cdca26407f0689127527dc364af4d34a87d8966cf123467d931bd6bac0491 not found: ID does not exist" containerID="b36cdca26407f0689127527dc364af4d34a87d8966cf123467d931bd6bac0491"
Aug 04 18:14:05.281984 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.281945    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36cdca26407f0689127527dc364af4d34a87d8966cf123467d931bd6bac0491"} err="failed to get container status \"b36cdca26407f0689127527dc364af4d34a87d8966cf123467d931bd6bac0491\": rpc error: code = NotFound desc = could not find container \"b36cdca26407f0689127527dc364af4d34a87d8966cf123467d931bd6bac0491\": container with ID starting with b36cdca26407f0689127527dc364af4d34a87d8966cf123467d931bd6bac0491 not found: ID does not exist"
Aug 04 18:14:05.281984 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.281962    2543 scope.go:117] "RemoveContainer" containerID="ff2b76beb1c02ff27526d17b2fc0cb9d09ce436a5cfc74efa42adae5593ba046"
Aug 04 18:14:05.282200 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:14:05.282180    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2b76beb1c02ff27526d17b2fc0cb9d09ce436a5cfc74efa42adae5593ba046\": container with ID starting with ff2b76beb1c02ff27526d17b2fc0cb9d09ce436a5cfc74efa42adae5593ba046 not found: ID does not exist" containerID="ff2b76beb1c02ff27526d17b2fc0cb9d09ce436a5cfc74efa42adae5593ba046"
Aug 04 18:14:05.282263 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.282210    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2b76beb1c02ff27526d17b2fc0cb9d09ce436a5cfc74efa42adae5593ba046"} err="failed to get container status \"ff2b76beb1c02ff27526d17b2fc0cb9d09ce436a5cfc74efa42adae5593ba046\": rpc error: code = NotFound desc = could not find container \"ff2b76beb1c02ff27526d17b2fc0cb9d09ce436a5cfc74efa42adae5593ba046\": container with ID starting with ff2b76beb1c02ff27526d17b2fc0cb9d09ce436a5cfc74efa42adae5593ba046 not found: ID does not exist"
Aug 04 18:14:05.282263 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.282233    2543 scope.go:117] "RemoveContainer" containerID="147962b3a758a2b5827c11bfb1afdfb345ddbe83d632e02ab91adf3a24bfe91b"
Aug 04 18:14:05.282481 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:14:05.282462    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147962b3a758a2b5827c11bfb1afdfb345ddbe83d632e02ab91adf3a24bfe91b\": container with ID starting with 147962b3a758a2b5827c11bfb1afdfb345ddbe83d632e02ab91adf3a24bfe91b not found: ID does not exist" containerID="147962b3a758a2b5827c11bfb1afdfb345ddbe83d632e02ab91adf3a24bfe91b"
Aug 04 18:14:05.282519 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.282486    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147962b3a758a2b5827c11bfb1afdfb345ddbe83d632e02ab91adf3a24bfe91b"} err="failed to get container status \"147962b3a758a2b5827c11bfb1afdfb345ddbe83d632e02ab91adf3a24bfe91b\": rpc error: code = NotFound desc = could not find container \"147962b3a758a2b5827c11bfb1afdfb345ddbe83d632e02ab91adf3a24bfe91b\": container with ID starting with 147962b3a758a2b5827c11bfb1afdfb345ddbe83d632e02ab91adf3a24bfe91b not found: ID does not exist"
Aug 04 18:14:05.282634 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.282617    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"]
Aug 04 18:14:05.286729 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.286707    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-9j6k4"]
Aug 04 18:14:05.574063 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:05.573984    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" path="/var/lib/kubelet/pods/754f4c06-cff3-4f1d-816f-2276676045ca/volumes"
Aug 04 18:14:16.053393 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.053361    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"]
Aug 04 18:14:16.053925 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.053659    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="kserve-container" containerID="cri-o://f4adae7689cffe2d0ee74ec70a1d610bb4e3f70565f744c6ed9b5087c1bac97b" gracePeriod=30
Aug 04 18:14:16.053925 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.053712    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="kube-rbac-proxy" containerID="cri-o://ddcbf66b2a53c5a409dde701afe96387930cf527a8f62566c46d5f0ccd650e02" gracePeriod=30
Aug 04 18:14:16.098174 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.098146    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"]
Aug 04 18:14:16.098461 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.098416    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kserve-container" containerID="cri-o://8ae52486e75d3c01435322721a339998138f2bcb882dc3fbe3138b3e27398be7" gracePeriod=30
Aug 04 18:14:16.098561 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.098435    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kube-rbac-proxy" containerID="cri-o://576190922cca686f452146a426251f1fd16df69ed5116435d4964c61d17ab579" gracePeriod=30
Aug 04 18:14:16.165389 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.165359    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"]
Aug 04 18:14:16.165826 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.165806    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent"
Aug 04 18:14:16.165826 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.165828    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent"
Aug 04 18:14:16.165959 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.165842    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kube-rbac-proxy"
Aug 04 18:14:16.165959 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.165851    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kube-rbac-proxy"
Aug 04 18:14:16.165959 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.165862    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container"
Aug 04 18:14:16.165959 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.165871    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container"
Aug 04 18:14:16.165959 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.165884    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kserve-container"
Aug 04 18:14:16.165959 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.165891    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kserve-container"
Aug 04 18:14:16.165959 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.165903    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="storage-initializer"
Aug 04 18:14:16.165959 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.165912    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="storage-initializer"
Aug 04 18:14:16.165959 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.165926    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="storage-initializer"
Aug 04 18:14:16.165959 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.165933    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="storage-initializer"
Aug 04 18:14:16.165959 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.165945    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kube-rbac-proxy"
Aug 04 18:14:16.165959 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.165953    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kube-rbac-proxy"
Aug 04 18:14:16.166328 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.166018    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kube-rbac-proxy"
Aug 04 18:14:16.166328 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.166028    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="754f4c06-cff3-4f1d-816f-2276676045ca" containerName="kserve-container"
Aug 04 18:14:16.166328 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.166036    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kserve-container"
Aug 04 18:14:16.166328 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.166041    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="agent"
Aug 04 18:14:16.166328 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.166047    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="936d63e2-5a19-4682-ac7e-6797a64783de" containerName="kube-rbac-proxy"
Aug 04 18:14:16.167830 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.167815    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.170496 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.170469    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-99f90-predictor-serving-cert\""
Aug 04 18:14:16.170596 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.170518    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-99f90-kube-rbac-proxy-sar-config\""
Aug 04 18:14:16.180790 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.180766    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"]
Aug 04 18:14:16.286819 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.286789    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"]
Aug 04 18:14:16.289360 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.289341    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.292137 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.292112    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-99f90-predictor-serving-cert\""
Aug 04 18:14:16.292230 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.292133    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-99f90-kube-rbac-proxy-sar-config\""
Aug 04 18:14:16.302143 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.302114    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"]
Aug 04 18:14:16.307254 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.307187    2543 generic.go:356] "Generic (PLEG): container finished" podID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerID="576190922cca686f452146a426251f1fd16df69ed5116435d4964c61d17ab579" exitCode=2
Aug 04 18:14:16.307254 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.307214    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" event={"ID":"b09ebb2d-7f40-4ebd-a850-c58d32199f96","Type":"ContainerDied","Data":"576190922cca686f452146a426251f1fd16df69ed5116435d4964c61d17ab579"}
Aug 04 18:14:16.309882 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.309859    2543 generic.go:356] "Generic (PLEG): container finished" podID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerID="ddcbf66b2a53c5a409dde701afe96387930cf527a8f62566c46d5f0ccd650e02" exitCode=2
Aug 04 18:14:16.309984 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.309892    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" event={"ID":"52069c82-27a0-4aeb-9be5-63288ef62e3b","Type":"ContainerDied","Data":"ddcbf66b2a53c5a409dde701afe96387930cf527a8f62566c46d5f0ccd650e02"}
Aug 04 18:14:16.330124 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.330100    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7355f5c-78a5-4cab-b175-f379f802516e-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.330238 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.330154    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7355f5c-78a5-4cab-b175-f379f802516e-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.330286 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.330234    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7355f5c-78a5-4cab-b175-f379f802516e-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.330356 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.330286    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7355f5c-78a5-4cab-b175-f379f802516e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.330356 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.330335    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9zqz\" (UniqueName: \"kubernetes.io/projected/d7355f5c-78a5-4cab-b175-f379f802516e-kube-api-access-f9zqz\") pod \"isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.431250 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.431217    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e74e25e-c752-4fcd-83d3-3c7dc26df936-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.431400 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.431261    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7355f5c-78a5-4cab-b175-f379f802516e-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.431400 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.431285    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.431400 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.431343    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7355f5c-78a5-4cab-b175-f379f802516e-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.431400 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.431386    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.431577 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.431435    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r77fb\" (UniqueName: \"kubernetes.io/projected/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kube-api-access-r77fb\") pod \"isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.431577 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.431500    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e74e25e-c752-4fcd-83d3-3c7dc26df936-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.431577 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.431545    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7355f5c-78a5-4cab-b175-f379f802516e-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.431712 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.431592    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7355f5c-78a5-4cab-b175-f379f802516e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.431712 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.431622    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9zqz\" (UniqueName: \"kubernetes.io/projected/d7355f5c-78a5-4cab-b175-f379f802516e-kube-api-access-f9zqz\") pod \"isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.431836 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.431753    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7355f5c-78a5-4cab-b175-f379f802516e-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.432081 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.432045    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7355f5c-78a5-4cab-b175-f379f802516e-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.432182 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.432165    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7355f5c-78a5-4cab-b175-f379f802516e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.433917 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.433895    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7355f5c-78a5-4cab-b175-f379f802516e-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.441051 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.441031    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9zqz\" (UniqueName: \"kubernetes.io/projected/d7355f5c-78a5-4cab-b175-f379f802516e-kube-api-access-f9zqz\") pod \"isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.478704 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.478681    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:16.533343 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.532592    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e74e25e-c752-4fcd-83d3-3c7dc26df936-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.533343 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.532651    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.533343 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.532707    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.533343 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.532753    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r77fb\" (UniqueName: \"kubernetes.io/projected/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kube-api-access-r77fb\") pod \"isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.533343 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.532812    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e74e25e-c752-4fcd-83d3-3c7dc26df936-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.533728 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.533524    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.533950 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.533904    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.534154 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.534134    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e74e25e-c752-4fcd-83d3-3c7dc26df936-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.535756 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.535728    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e74e25e-c752-4fcd-83d3-3c7dc26df936-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.542101 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.542080    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r77fb\" (UniqueName: \"kubernetes.io/projected/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kube-api-access-r77fb\") pod \"isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.600816 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.600768    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:16.616049 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.616028    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"]
Aug 04 18:14:16.617952 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:14:16.617930    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7355f5c_78a5_4cab_b175_f379f802516e.slice/crio-ce9239037d2e5034b8f4ac88a4a0fd9fc069186f12c4253005969ac049f7c1d8 WatchSource:0}: Error finding container ce9239037d2e5034b8f4ac88a4a0fd9fc069186f12c4253005969ac049f7c1d8: Status 404 returned error can't find the container with id ce9239037d2e5034b8f4ac88a4a0fd9fc069186f12c4253005969ac049f7c1d8
Aug 04 18:14:16.738808 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:16.738784    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"]
Aug 04 18:14:16.740556 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:14:16.740532    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e74e25e_c752_4fcd_83d3_3c7dc26df936.slice/crio-4f60f1fbd8c769526b94e0076f8933466cbfe083ace06222bad26b968f9df0ef WatchSource:0}: Error finding container 4f60f1fbd8c769526b94e0076f8933466cbfe083ace06222bad26b968f9df0ef: Status 404 returned error can't find the container with id 4f60f1fbd8c769526b94e0076f8933466cbfe083ace06222bad26b968f9df0ef
Aug 04 18:14:17.315925 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:17.315887    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" event={"ID":"1e74e25e-c752-4fcd-83d3-3c7dc26df936","Type":"ContainerStarted","Data":"8e2520e195be85e9606104f97bbf4eaa147c0117235572981a0b90607b076883"}
Aug 04 18:14:17.316542 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:17.316508    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" event={"ID":"1e74e25e-c752-4fcd-83d3-3c7dc26df936","Type":"ContainerStarted","Data":"4f60f1fbd8c769526b94e0076f8933466cbfe083ace06222bad26b968f9df0ef"}
Aug 04 18:14:17.317695 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:17.317670    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" event={"ID":"d7355f5c-78a5-4cab-b175-f379f802516e","Type":"ContainerStarted","Data":"4c3b1dfd61a992e543dd5012f2ab8583f01bdad84b85ab072665517aca485ff7"}
Aug 04 18:14:17.317782 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:17.317737    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" event={"ID":"d7355f5c-78a5-4cab-b175-f379f802516e","Type":"ContainerStarted","Data":"ce9239037d2e5034b8f4ac88a4a0fd9fc069186f12c4253005969ac049f7c1d8"}
Aug 04 18:14:18.847958 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:18.847920    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Aug 04 18:14:18.854535 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:18.854499    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:14:19.699446 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.699415    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:14:19.764917 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.764838    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj2n2\" (UniqueName: \"kubernetes.io/projected/52069c82-27a0-4aeb-9be5-63288ef62e3b-kube-api-access-kj2n2\") pod \"52069c82-27a0-4aeb-9be5-63288ef62e3b\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") "
Aug 04 18:14:19.764917 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.764876    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52069c82-27a0-4aeb-9be5-63288ef62e3b-kserve-provision-location\") pod \"52069c82-27a0-4aeb-9be5-63288ef62e3b\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") "
Aug 04 18:14:19.764917 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.764912    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52069c82-27a0-4aeb-9be5-63288ef62e3b-kube-rbac-proxy-sar-config\") pod \"52069c82-27a0-4aeb-9be5-63288ef62e3b\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") "
Aug 04 18:14:19.765175 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.764950    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52069c82-27a0-4aeb-9be5-63288ef62e3b-proxy-tls\") pod \"52069c82-27a0-4aeb-9be5-63288ef62e3b\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") "
Aug 04 18:14:19.765175 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.764969    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52069c82-27a0-4aeb-9be5-63288ef62e3b-cabundle-cert\") pod \"52069c82-27a0-4aeb-9be5-63288ef62e3b\" (UID: \"52069c82-27a0-4aeb-9be5-63288ef62e3b\") "
Aug 04 18:14:19.765282 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.765211    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52069c82-27a0-4aeb-9be5-63288ef62e3b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52069c82-27a0-4aeb-9be5-63288ef62e3b" (UID: "52069c82-27a0-4aeb-9be5-63288ef62e3b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:19.765415 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.765389    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52069c82-27a0-4aeb-9be5-63288ef62e3b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "52069c82-27a0-4aeb-9be5-63288ef62e3b" (UID: "52069c82-27a0-4aeb-9be5-63288ef62e3b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:14:19.765483 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.765461    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52069c82-27a0-4aeb-9be5-63288ef62e3b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "52069c82-27a0-4aeb-9be5-63288ef62e3b" (UID: "52069c82-27a0-4aeb-9be5-63288ef62e3b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:14:19.766977 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.766955    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52069c82-27a0-4aeb-9be5-63288ef62e3b-kube-api-access-kj2n2" (OuterVolumeSpecName: "kube-api-access-kj2n2") pod "52069c82-27a0-4aeb-9be5-63288ef62e3b" (UID: "52069c82-27a0-4aeb-9be5-63288ef62e3b"). InnerVolumeSpecName "kube-api-access-kj2n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:14:19.767043 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.767011    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52069c82-27a0-4aeb-9be5-63288ef62e3b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "52069c82-27a0-4aeb-9be5-63288ef62e3b" (UID: "52069c82-27a0-4aeb-9be5-63288ef62e3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:14:19.865695 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.865667    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj2n2\" (UniqueName: \"kubernetes.io/projected/52069c82-27a0-4aeb-9be5-63288ef62e3b-kube-api-access-kj2n2\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:14:19.865695 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.865692    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52069c82-27a0-4aeb-9be5-63288ef62e3b-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:14:19.865695 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.865701    2543 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52069c82-27a0-4aeb-9be5-63288ef62e3b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:14:19.866081 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.865711    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52069c82-27a0-4aeb-9be5-63288ef62e3b-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:14:19.866081 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:19.865722    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52069c82-27a0-4aeb-9be5-63288ef62e3b-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:14:20.334095 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.334062    2543 generic.go:356] "Generic (PLEG): container finished" podID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerID="f4adae7689cffe2d0ee74ec70a1d610bb4e3f70565f744c6ed9b5087c1bac97b" exitCode=0
Aug 04 18:14:20.334274 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.334116    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" event={"ID":"52069c82-27a0-4aeb-9be5-63288ef62e3b","Type":"ContainerDied","Data":"f4adae7689cffe2d0ee74ec70a1d610bb4e3f70565f744c6ed9b5087c1bac97b"}
Aug 04 18:14:20.334274 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.334160    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch" event={"ID":"52069c82-27a0-4aeb-9be5-63288ef62e3b","Type":"ContainerDied","Data":"8781b1e1c80d0e94c26a25375e66eebc31047ad1c22180f722198fe709a75984"}
Aug 04 18:14:20.334274 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.334186    2543 scope.go:117] "RemoveContainer" containerID="ddcbf66b2a53c5a409dde701afe96387930cf527a8f62566c46d5f0ccd650e02"
Aug 04 18:14:20.334274 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.334116    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"
Aug 04 18:14:20.342728 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.342713    2543 scope.go:117] "RemoveContainer" containerID="f4adae7689cffe2d0ee74ec70a1d610bb4e3f70565f744c6ed9b5087c1bac97b"
Aug 04 18:14:20.350102 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.350087    2543 scope.go:117] "RemoveContainer" containerID="fdcf227031f979f26aac37b06021959782ecf3f5feb5741e30c21ed6088cd466"
Aug 04 18:14:20.358261 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.358246    2543 scope.go:117] "RemoveContainer" containerID="ddcbf66b2a53c5a409dde701afe96387930cf527a8f62566c46d5f0ccd650e02"
Aug 04 18:14:20.358416 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.358392    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"]
Aug 04 18:14:20.358584 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:14:20.358564    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddcbf66b2a53c5a409dde701afe96387930cf527a8f62566c46d5f0ccd650e02\": container with ID starting with ddcbf66b2a53c5a409dde701afe96387930cf527a8f62566c46d5f0ccd650e02 not found: ID does not exist" containerID="ddcbf66b2a53c5a409dde701afe96387930cf527a8f62566c46d5f0ccd650e02"
Aug 04 18:14:20.358662 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.358597    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddcbf66b2a53c5a409dde701afe96387930cf527a8f62566c46d5f0ccd650e02"} err="failed to get container status \"ddcbf66b2a53c5a409dde701afe96387930cf527a8f62566c46d5f0ccd650e02\": rpc error: code = NotFound desc = could not find container \"ddcbf66b2a53c5a409dde701afe96387930cf527a8f62566c46d5f0ccd650e02\": container with ID starting with ddcbf66b2a53c5a409dde701afe96387930cf527a8f62566c46d5f0ccd650e02 not found: ID does not exist"
Aug 04 18:14:20.358662 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.358622    2543 scope.go:117] "RemoveContainer" containerID="f4adae7689cffe2d0ee74ec70a1d610bb4e3f70565f744c6ed9b5087c1bac97b"
Aug 04 18:14:20.358909 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:14:20.358891    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4adae7689cffe2d0ee74ec70a1d610bb4e3f70565f744c6ed9b5087c1bac97b\": container with ID starting with f4adae7689cffe2d0ee74ec70a1d610bb4e3f70565f744c6ed9b5087c1bac97b not found: ID does not exist" containerID="f4adae7689cffe2d0ee74ec70a1d610bb4e3f70565f744c6ed9b5087c1bac97b"
Aug 04 18:14:20.358950 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.358916    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4adae7689cffe2d0ee74ec70a1d610bb4e3f70565f744c6ed9b5087c1bac97b"} err="failed to get container status \"f4adae7689cffe2d0ee74ec70a1d610bb4e3f70565f744c6ed9b5087c1bac97b\": rpc error: code = NotFound desc = could not find container \"f4adae7689cffe2d0ee74ec70a1d610bb4e3f70565f744c6ed9b5087c1bac97b\": container with ID starting with f4adae7689cffe2d0ee74ec70a1d610bb4e3f70565f744c6ed9b5087c1bac97b not found: ID does not exist"
Aug 04 18:14:20.358950 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.358934    2543 scope.go:117] "RemoveContainer" containerID="fdcf227031f979f26aac37b06021959782ecf3f5feb5741e30c21ed6088cd466"
Aug 04 18:14:20.359156 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:14:20.359138    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdcf227031f979f26aac37b06021959782ecf3f5feb5741e30c21ed6088cd466\": container with ID starting with fdcf227031f979f26aac37b06021959782ecf3f5feb5741e30c21ed6088cd466 not found: ID does not exist" containerID="fdcf227031f979f26aac37b06021959782ecf3f5feb5741e30c21ed6088cd466"
Aug 04 18:14:20.359215 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.359163    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdcf227031f979f26aac37b06021959782ecf3f5feb5741e30c21ed6088cd466"} err="failed to get container status \"fdcf227031f979f26aac37b06021959782ecf3f5feb5741e30c21ed6088cd466\": rpc error: code = NotFound desc = could not find container \"fdcf227031f979f26aac37b06021959782ecf3f5feb5741e30c21ed6088cd466\": container with ID starting with fdcf227031f979f26aac37b06021959782ecf3f5feb5741e30c21ed6088cd466 not found: ID does not exist"
Aug 04 18:14:20.366297 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.366275    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c55f6-predictor-5d988d489-45lch"]
Aug 04 18:14:20.939674 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.939650    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:14:20.975419 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.975396    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq2wc\" (UniqueName: \"kubernetes.io/projected/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kube-api-access-jq2wc\") pod \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") "
Aug 04 18:14:20.975539 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.975460    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b09ebb2d-7f40-4ebd-a850-c58d32199f96-cabundle-cert\") pod \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") "
Aug 04 18:14:20.975539 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.975523    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kserve-provision-location\") pod \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") "
Aug 04 18:14:20.975624 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.975548    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kube-rbac-proxy-sar-config\") pod \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") "
Aug 04 18:14:20.975624 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.975574    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b09ebb2d-7f40-4ebd-a850-c58d32199f96-proxy-tls\") pod \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\" (UID: \"b09ebb2d-7f40-4ebd-a850-c58d32199f96\") "
Aug 04 18:14:20.975856 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.975827    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b09ebb2d-7f40-4ebd-a850-c58d32199f96" (UID: "b09ebb2d-7f40-4ebd-a850-c58d32199f96"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:20.975952 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.975882    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09ebb2d-7f40-4ebd-a850-c58d32199f96-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b09ebb2d-7f40-4ebd-a850-c58d32199f96" (UID: "b09ebb2d-7f40-4ebd-a850-c58d32199f96"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:14:20.975952 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.975904    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b09ebb2d-7f40-4ebd-a850-c58d32199f96" (UID: "b09ebb2d-7f40-4ebd-a850-c58d32199f96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:14:20.977622 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.977595    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b09ebb2d-7f40-4ebd-a850-c58d32199f96-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b09ebb2d-7f40-4ebd-a850-c58d32199f96" (UID: "b09ebb2d-7f40-4ebd-a850-c58d32199f96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:14:20.977697 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:20.977653    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kube-api-access-jq2wc" (OuterVolumeSpecName: "kube-api-access-jq2wc") pod "b09ebb2d-7f40-4ebd-a850-c58d32199f96" (UID: "b09ebb2d-7f40-4ebd-a850-c58d32199f96"). InnerVolumeSpecName "kube-api-access-jq2wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:14:21.076983 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.076913    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b09ebb2d-7f40-4ebd-a850-c58d32199f96-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:14:21.076983 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.076934    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:14:21.076983 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.076943    2543 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:14:21.076983 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.076952    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b09ebb2d-7f40-4ebd-a850-c58d32199f96-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:14:21.076983 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.076960    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq2wc\" (UniqueName: \"kubernetes.io/projected/b09ebb2d-7f40-4ebd-a850-c58d32199f96-kube-api-access-jq2wc\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:14:21.339461 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.339372    2543 generic.go:356] "Generic (PLEG): container finished" podID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerID="8e2520e195be85e9606104f97bbf4eaa147c0117235572981a0b90607b076883" exitCode=0
Aug 04 18:14:21.339727 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.339451    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" event={"ID":"1e74e25e-c752-4fcd-83d3-3c7dc26df936","Type":"ContainerDied","Data":"8e2520e195be85e9606104f97bbf4eaa147c0117235572981a0b90607b076883"}
Aug 04 18:14:21.340993 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.340967    2543 generic.go:356] "Generic (PLEG): container finished" podID="d7355f5c-78a5-4cab-b175-f379f802516e" containerID="4c3b1dfd61a992e543dd5012f2ab8583f01bdad84b85ab072665517aca485ff7" exitCode=0
Aug 04 18:14:21.341112 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.341040    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" event={"ID":"d7355f5c-78a5-4cab-b175-f379f802516e","Type":"ContainerDied","Data":"4c3b1dfd61a992e543dd5012f2ab8583f01bdad84b85ab072665517aca485ff7"}
Aug 04 18:14:21.343905 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.343884    2543 generic.go:356] "Generic (PLEG): container finished" podID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerID="8ae52486e75d3c01435322721a339998138f2bcb882dc3fbe3138b3e27398be7" exitCode=0
Aug 04 18:14:21.343986 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.343920    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" event={"ID":"b09ebb2d-7f40-4ebd-a850-c58d32199f96","Type":"ContainerDied","Data":"8ae52486e75d3c01435322721a339998138f2bcb882dc3fbe3138b3e27398be7"}
Aug 04 18:14:21.343986 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.343932    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"
Aug 04 18:14:21.343986 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.343973    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn" event={"ID":"b09ebb2d-7f40-4ebd-a850-c58d32199f96","Type":"ContainerDied","Data":"db7649dcffd04982ccbd2e5e9e2fab4f9be717b800d6311e0a1e23c60191d3aa"}
Aug 04 18:14:21.344125 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.344010    2543 scope.go:117] "RemoveContainer" containerID="576190922cca686f452146a426251f1fd16df69ed5116435d4964c61d17ab579"
Aug 04 18:14:21.352716 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.352689    2543 scope.go:117] "RemoveContainer" containerID="8ae52486e75d3c01435322721a339998138f2bcb882dc3fbe3138b3e27398be7"
Aug 04 18:14:21.362961 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.362941    2543 scope.go:117] "RemoveContainer" containerID="4b0902decba1ef099affb424231792f379c751ba9efe4b66602e894c235c8ec9"
Aug 04 18:14:21.374825 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.374803    2543 scope.go:117] "RemoveContainer" containerID="576190922cca686f452146a426251f1fd16df69ed5116435d4964c61d17ab579"
Aug 04 18:14:21.375106 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:14:21.375086    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"576190922cca686f452146a426251f1fd16df69ed5116435d4964c61d17ab579\": container with ID starting with 576190922cca686f452146a426251f1fd16df69ed5116435d4964c61d17ab579 not found: ID does not exist" containerID="576190922cca686f452146a426251f1fd16df69ed5116435d4964c61d17ab579"
Aug 04 18:14:21.375179 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.375113    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576190922cca686f452146a426251f1fd16df69ed5116435d4964c61d17ab579"} err="failed to get container status \"576190922cca686f452146a426251f1fd16df69ed5116435d4964c61d17ab579\": rpc error: code = NotFound desc = could not find container \"576190922cca686f452146a426251f1fd16df69ed5116435d4964c61d17ab579\": container with ID starting with 576190922cca686f452146a426251f1fd16df69ed5116435d4964c61d17ab579 not found: ID does not exist"
Aug 04 18:14:21.375179 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.375130    2543 scope.go:117] "RemoveContainer" containerID="8ae52486e75d3c01435322721a339998138f2bcb882dc3fbe3138b3e27398be7"
Aug 04 18:14:21.375380 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:14:21.375343    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae52486e75d3c01435322721a339998138f2bcb882dc3fbe3138b3e27398be7\": container with ID starting with 8ae52486e75d3c01435322721a339998138f2bcb882dc3fbe3138b3e27398be7 not found: ID does not exist" containerID="8ae52486e75d3c01435322721a339998138f2bcb882dc3fbe3138b3e27398be7"
Aug 04 18:14:21.375380 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.375364    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae52486e75d3c01435322721a339998138f2bcb882dc3fbe3138b3e27398be7"} err="failed to get container status \"8ae52486e75d3c01435322721a339998138f2bcb882dc3fbe3138b3e27398be7\": rpc error: code = NotFound desc = could not find container \"8ae52486e75d3c01435322721a339998138f2bcb882dc3fbe3138b3e27398be7\": container with ID starting with 8ae52486e75d3c01435322721a339998138f2bcb882dc3fbe3138b3e27398be7 not found: ID does not exist"
Aug 04 18:14:21.375380 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.375377    2543 scope.go:117] "RemoveContainer" containerID="4b0902decba1ef099affb424231792f379c751ba9efe4b66602e894c235c8ec9"
Aug 04 18:14:21.375602 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:14:21.375586    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b0902decba1ef099affb424231792f379c751ba9efe4b66602e894c235c8ec9\": container with ID starting with 4b0902decba1ef099affb424231792f379c751ba9efe4b66602e894c235c8ec9 not found: ID does not exist" containerID="4b0902decba1ef099affb424231792f379c751ba9efe4b66602e894c235c8ec9"
Aug 04 18:14:21.375641 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.375606    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b0902decba1ef099affb424231792f379c751ba9efe4b66602e894c235c8ec9"} err="failed to get container status \"4b0902decba1ef099affb424231792f379c751ba9efe4b66602e894c235c8ec9\": rpc error: code = NotFound desc = could not find container \"4b0902decba1ef099affb424231792f379c751ba9efe4b66602e894c235c8ec9\": container with ID starting with 4b0902decba1ef099affb424231792f379c751ba9efe4b66602e894c235c8ec9 not found: ID does not exist"
Aug 04 18:14:21.376655 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.376636    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"]
Aug 04 18:14:21.382919 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.382896    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c55f6-predictor-8bcb659bc-k9qmn"]
Aug 04 18:14:21.575219 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.575189    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" path="/var/lib/kubelet/pods/52069c82-27a0-4aeb-9be5-63288ef62e3b/volumes"
Aug 04 18:14:21.575684 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:21.575669    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" path="/var/lib/kubelet/pods/b09ebb2d-7f40-4ebd-a850-c58d32199f96/volumes"
Aug 04 18:14:22.351798 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:22.351767    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" event={"ID":"1e74e25e-c752-4fcd-83d3-3c7dc26df936","Type":"ContainerStarted","Data":"9454f50ebf4b94d097aa57cbb4443c933892b94d248e119b0bb4d7a97061fce1"}
Aug 04 18:14:22.352229 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:22.351823    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" event={"ID":"1e74e25e-c752-4fcd-83d3-3c7dc26df936","Type":"ContainerStarted","Data":"99cc7b811eb2caf722110f99039d2f883afde23527cddf0bb2bfea8648302e48"}
Aug 04 18:14:22.352229 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:22.352025    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:22.354060 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:22.354036    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" event={"ID":"d7355f5c-78a5-4cab-b175-f379f802516e","Type":"ContainerStarted","Data":"db96fc5357dbaafbaab1d37fc507c0244ef9503cb9e5e2e8c1efa7b57b0aa888"}
Aug 04 18:14:22.354186 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:22.354085    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" event={"ID":"d7355f5c-78a5-4cab-b175-f379f802516e","Type":"ContainerStarted","Data":"a21d0951b1be44a150db3b6492a9ffdf02083f9126b47078cc077e3c15ecb5cc"}
Aug 04 18:14:22.354435 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:22.354402    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:22.354538 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:22.354460    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:22.355936 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:22.355909    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:14:22.373326 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:22.373271    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" podStartSLOduration=2.660227205 podStartE2EDuration="6.373256701s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.713029496s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:14:22.370765021 +0000 UTC m=+865.341409387" watchObservedRunningTime="2026-08-04 18:14:22.373256701 +0000 UTC m=+865.343901042"
Aug 04 18:14:22.390150 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:22.390112    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" podStartSLOduration=2.632136052 podStartE2EDuration="6.390098719s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.757962667s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:14:22.388092059 +0000 UTC m=+865.358736401" watchObservedRunningTime="2026-08-04 18:14:22.390098719 +0000 UTC m=+865.360743123"
Aug 04 18:14:23.360510 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:23.360461    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:14:23.360989 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:23.360713    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:23.362250 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:23.362215    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:14:24.365270 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:24.365229    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:14:28.365462 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:28.365428    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:14:28.365881 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:28.365855    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:14:29.369715 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:29.369688    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:14:29.370271 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:29.370244    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:14:38.366802 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:38.366756    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:14:39.370165 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:39.370128    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:14:48.366088 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:48.366048    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:14:49.370256 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:49.370211    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:14:57.497716 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:57.497643    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/1.log"
Aug 04 18:14:57.499048 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:57.499022    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/1.log"
Aug 04 18:14:58.366785 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:58.366745    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:14:59.371163 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:14:59.371111    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:15:08.366799 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:15:08.366761    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:15:09.370804 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:15:09.370765    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:15:18.366070 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:15:18.366031    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:15:19.370824 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:15:19.370797    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:15:28.365940 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:15:28.365898    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:15:31.570043 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:15:31.570004    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:15:41.573650 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:15:41.573623    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:16:06.507784 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:06.507753    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"]
Aug 04 18:16:06.510520 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:06.508018    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kserve-container" containerID="cri-o://a21d0951b1be44a150db3b6492a9ffdf02083f9126b47078cc077e3c15ecb5cc" gracePeriod=30
Aug 04 18:16:06.510520 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:06.508059    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kube-rbac-proxy" containerID="cri-o://db96fc5357dbaafbaab1d37fc507c0244ef9503cb9e5e2e8c1efa7b57b0aa888" gracePeriod=30
Aug 04 18:16:06.661588 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:06.661559    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"]
Aug 04 18:16:06.661797 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:06.661770    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kserve-container" containerID="cri-o://99cc7b811eb2caf722110f99039d2f883afde23527cddf0bb2bfea8648302e48" gracePeriod=30
Aug 04 18:16:06.661934 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:06.661824    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kube-rbac-proxy" containerID="cri-o://9454f50ebf4b94d097aa57cbb4443c933892b94d248e119b0bb4d7a97061fce1" gracePeriod=30
Aug 04 18:16:06.782639 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:06.782612    2543 generic.go:356] "Generic (PLEG): container finished" podID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerID="9454f50ebf4b94d097aa57cbb4443c933892b94d248e119b0bb4d7a97061fce1" exitCode=2
Aug 04 18:16:06.782773 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:06.782682    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" event={"ID":"1e74e25e-c752-4fcd-83d3-3c7dc26df936","Type":"ContainerDied","Data":"9454f50ebf4b94d097aa57cbb4443c933892b94d248e119b0bb4d7a97061fce1"}
Aug 04 18:16:06.784537 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:06.784518    2543 generic.go:356] "Generic (PLEG): container finished" podID="d7355f5c-78a5-4cab-b175-f379f802516e" containerID="db96fc5357dbaafbaab1d37fc507c0244ef9503cb9e5e2e8c1efa7b57b0aa888" exitCode=2
Aug 04 18:16:06.784627 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:06.784586    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" event={"ID":"d7355f5c-78a5-4cab-b175-f379f802516e","Type":"ContainerDied","Data":"db96fc5357dbaafbaab1d37fc507c0244ef9503cb9e5e2e8c1efa7b57b0aa888"}
Aug 04 18:16:08.361302 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:08.361254    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Aug 04 18:16:09.366207 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:09.366163    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Aug 04 18:16:09.370573 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:09.370538    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:16:10.005196 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.005171    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:16:10.086985 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.086954    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e74e25e-c752-4fcd-83d3-3c7dc26df936-proxy-tls\") pod \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") "
Aug 04 18:16:10.087164 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.087002    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kube-rbac-proxy-sar-config\") pod \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") "
Aug 04 18:16:10.087164 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.087018    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e74e25e-c752-4fcd-83d3-3c7dc26df936-cabundle-cert\") pod \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") "
Aug 04 18:16:10.087164 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.087090    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r77fb\" (UniqueName: \"kubernetes.io/projected/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kube-api-access-r77fb\") pod \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") "
Aug 04 18:16:10.087164 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.087122    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kserve-provision-location\") pod \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\" (UID: \"1e74e25e-c752-4fcd-83d3-3c7dc26df936\") "
Aug 04 18:16:10.087504 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.087465    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1e74e25e-c752-4fcd-83d3-3c7dc26df936" (UID: "1e74e25e-c752-4fcd-83d3-3c7dc26df936"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:16:10.087504 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.087480    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e74e25e-c752-4fcd-83d3-3c7dc26df936-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1e74e25e-c752-4fcd-83d3-3c7dc26df936" (UID: "1e74e25e-c752-4fcd-83d3-3c7dc26df936"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:16:10.087644 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.087525    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1e74e25e-c752-4fcd-83d3-3c7dc26df936" (UID: "1e74e25e-c752-4fcd-83d3-3c7dc26df936"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:16:10.088970 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.088947    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e74e25e-c752-4fcd-83d3-3c7dc26df936-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e74e25e-c752-4fcd-83d3-3c7dc26df936" (UID: "1e74e25e-c752-4fcd-83d3-3c7dc26df936"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:16:10.089138 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.089121    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kube-api-access-r77fb" (OuterVolumeSpecName: "kube-api-access-r77fb") pod "1e74e25e-c752-4fcd-83d3-3c7dc26df936" (UID: "1e74e25e-c752-4fcd-83d3-3c7dc26df936"). InnerVolumeSpecName "kube-api-access-r77fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:16:10.187847 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.187755    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r77fb\" (UniqueName: \"kubernetes.io/projected/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kube-api-access-r77fb\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:16:10.187847 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.187793    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:16:10.187847 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.187804    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e74e25e-c752-4fcd-83d3-3c7dc26df936-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:16:10.187847 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.187813    2543 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e74e25e-c752-4fcd-83d3-3c7dc26df936-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:16:10.187847 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.187822    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e74e25e-c752-4fcd-83d3-3c7dc26df936-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:16:10.801685 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.801656    2543 generic.go:356] "Generic (PLEG): container finished" podID="d7355f5c-78a5-4cab-b175-f379f802516e" containerID="a21d0951b1be44a150db3b6492a9ffdf02083f9126b47078cc077e3c15ecb5cc" exitCode=0
Aug 04 18:16:10.802045 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.801734    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" event={"ID":"d7355f5c-78a5-4cab-b175-f379f802516e","Type":"ContainerDied","Data":"a21d0951b1be44a150db3b6492a9ffdf02083f9126b47078cc077e3c15ecb5cc"}
Aug 04 18:16:10.803804 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.803781    2543 generic.go:356] "Generic (PLEG): container finished" podID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerID="99cc7b811eb2caf722110f99039d2f883afde23527cddf0bb2bfea8648302e48" exitCode=0
Aug 04 18:16:10.803901 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.803818    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" event={"ID":"1e74e25e-c752-4fcd-83d3-3c7dc26df936","Type":"ContainerDied","Data":"99cc7b811eb2caf722110f99039d2f883afde23527cddf0bb2bfea8648302e48"}
Aug 04 18:16:10.803901 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.803846    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"
Aug 04 18:16:10.803901 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.803880    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww" event={"ID":"1e74e25e-c752-4fcd-83d3-3c7dc26df936","Type":"ContainerDied","Data":"4f60f1fbd8c769526b94e0076f8933466cbfe083ace06222bad26b968f9df0ef"}
Aug 04 18:16:10.804015 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.803923    2543 scope.go:117] "RemoveContainer" containerID="9454f50ebf4b94d097aa57cbb4443c933892b94d248e119b0bb4d7a97061fce1"
Aug 04 18:16:10.812361 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.812342    2543 scope.go:117] "RemoveContainer" containerID="99cc7b811eb2caf722110f99039d2f883afde23527cddf0bb2bfea8648302e48"
Aug 04 18:16:10.819841 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.819827    2543 scope.go:117] "RemoveContainer" containerID="8e2520e195be85e9606104f97bbf4eaa147c0117235572981a0b90607b076883"
Aug 04 18:16:10.827553 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.827534    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"]
Aug 04 18:16:10.827964 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.827940    2543 scope.go:117] "RemoveContainer" containerID="9454f50ebf4b94d097aa57cbb4443c933892b94d248e119b0bb4d7a97061fce1"
Aug 04 18:16:10.828225 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:16:10.828203    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9454f50ebf4b94d097aa57cbb4443c933892b94d248e119b0bb4d7a97061fce1\": container with ID starting with 9454f50ebf4b94d097aa57cbb4443c933892b94d248e119b0bb4d7a97061fce1 not found: ID does not exist" containerID="9454f50ebf4b94d097aa57cbb4443c933892b94d248e119b0bb4d7a97061fce1"
Aug 04 18:16:10.828283 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.828233    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9454f50ebf4b94d097aa57cbb4443c933892b94d248e119b0bb4d7a97061fce1"} err="failed to get container status \"9454f50ebf4b94d097aa57cbb4443c933892b94d248e119b0bb4d7a97061fce1\": rpc error: code = NotFound desc = could not find container \"9454f50ebf4b94d097aa57cbb4443c933892b94d248e119b0bb4d7a97061fce1\": container with ID starting with 9454f50ebf4b94d097aa57cbb4443c933892b94d248e119b0bb4d7a97061fce1 not found: ID does not exist"
Aug 04 18:16:10.828283 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.828258    2543 scope.go:117] "RemoveContainer" containerID="99cc7b811eb2caf722110f99039d2f883afde23527cddf0bb2bfea8648302e48"
Aug 04 18:16:10.828508 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:16:10.828492    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99cc7b811eb2caf722110f99039d2f883afde23527cddf0bb2bfea8648302e48\": container with ID starting with 99cc7b811eb2caf722110f99039d2f883afde23527cddf0bb2bfea8648302e48 not found: ID does not exist" containerID="99cc7b811eb2caf722110f99039d2f883afde23527cddf0bb2bfea8648302e48"
Aug 04 18:16:10.828555 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.828513    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99cc7b811eb2caf722110f99039d2f883afde23527cddf0bb2bfea8648302e48"} err="failed to get container status \"99cc7b811eb2caf722110f99039d2f883afde23527cddf0bb2bfea8648302e48\": rpc error: code = NotFound desc = could not find container \"99cc7b811eb2caf722110f99039d2f883afde23527cddf0bb2bfea8648302e48\": container with ID starting with 99cc7b811eb2caf722110f99039d2f883afde23527cddf0bb2bfea8648302e48 not found: ID does not exist"
Aug 04 18:16:10.828555 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.828529    2543 scope.go:117] "RemoveContainer" containerID="8e2520e195be85e9606104f97bbf4eaa147c0117235572981a0b90607b076883"
Aug 04 18:16:10.828803 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:16:10.828786    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2520e195be85e9606104f97bbf4eaa147c0117235572981a0b90607b076883\": container with ID starting with 8e2520e195be85e9606104f97bbf4eaa147c0117235572981a0b90607b076883 not found: ID does not exist" containerID="8e2520e195be85e9606104f97bbf4eaa147c0117235572981a0b90607b076883"
Aug 04 18:16:10.828860 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.828824    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2520e195be85e9606104f97bbf4eaa147c0117235572981a0b90607b076883"} err="failed to get container status \"8e2520e195be85e9606104f97bbf4eaa147c0117235572981a0b90607b076883\": rpc error: code = NotFound desc = could not find container \"8e2520e195be85e9606104f97bbf4eaa147c0117235572981a0b90607b076883\": container with ID starting with 8e2520e195be85e9606104f97bbf4eaa147c0117235572981a0b90607b076883 not found: ID does not exist"
Aug 04 18:16:10.832224 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.832201    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-99f90-predictor-5b7cf45cc-xtbww"]
Aug 04 18:16:10.843192 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.843176    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:16:10.895697 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.895672    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7355f5c-78a5-4cab-b175-f379f802516e-cabundle-cert\") pod \"d7355f5c-78a5-4cab-b175-f379f802516e\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") "
Aug 04 18:16:10.895790 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.895721    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7355f5c-78a5-4cab-b175-f379f802516e-kube-rbac-proxy-sar-config\") pod \"d7355f5c-78a5-4cab-b175-f379f802516e\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") "
Aug 04 18:16:10.895790 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.895746    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7355f5c-78a5-4cab-b175-f379f802516e-proxy-tls\") pod \"d7355f5c-78a5-4cab-b175-f379f802516e\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") "
Aug 04 18:16:10.895790 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.895772    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7355f5c-78a5-4cab-b175-f379f802516e-kserve-provision-location\") pod \"d7355f5c-78a5-4cab-b175-f379f802516e\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") "
Aug 04 18:16:10.895951 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.895807    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9zqz\" (UniqueName: \"kubernetes.io/projected/d7355f5c-78a5-4cab-b175-f379f802516e-kube-api-access-f9zqz\") pod \"d7355f5c-78a5-4cab-b175-f379f802516e\" (UID: \"d7355f5c-78a5-4cab-b175-f379f802516e\") "
Aug 04 18:16:10.896024 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.895999    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7355f5c-78a5-4cab-b175-f379f802516e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d7355f5c-78a5-4cab-b175-f379f802516e" (UID: "d7355f5c-78a5-4cab-b175-f379f802516e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:16:10.896144 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.896122    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7355f5c-78a5-4cab-b175-f379f802516e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d7355f5c-78a5-4cab-b175-f379f802516e" (UID: "d7355f5c-78a5-4cab-b175-f379f802516e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:16:10.896197 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.896140    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7355f5c-78a5-4cab-b175-f379f802516e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d7355f5c-78a5-4cab-b175-f379f802516e" (UID: "d7355f5c-78a5-4cab-b175-f379f802516e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:16:10.897838 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.897818    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7355f5c-78a5-4cab-b175-f379f802516e-kube-api-access-f9zqz" (OuterVolumeSpecName: "kube-api-access-f9zqz") pod "d7355f5c-78a5-4cab-b175-f379f802516e" (UID: "d7355f5c-78a5-4cab-b175-f379f802516e"). InnerVolumeSpecName "kube-api-access-f9zqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:16:10.897887 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.897843    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7355f5c-78a5-4cab-b175-f379f802516e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d7355f5c-78a5-4cab-b175-f379f802516e" (UID: "d7355f5c-78a5-4cab-b175-f379f802516e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:16:10.997164 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.997110    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7355f5c-78a5-4cab-b175-f379f802516e-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:16:10.997164 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.997132    2543 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7355f5c-78a5-4cab-b175-f379f802516e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:16:10.997164 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.997142    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7355f5c-78a5-4cab-b175-f379f802516e-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:16:10.997164 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.997150    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7355f5c-78a5-4cab-b175-f379f802516e-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:16:10.997164 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:10.997158    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9zqz\" (UniqueName: \"kubernetes.io/projected/d7355f5c-78a5-4cab-b175-f379f802516e-kube-api-access-f9zqz\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:16:11.573862 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:11.573827    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" path="/var/lib/kubelet/pods/1e74e25e-c752-4fcd-83d3-3c7dc26df936/volumes"
Aug 04 18:16:11.809963 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:11.809910    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv" event={"ID":"d7355f5c-78a5-4cab-b175-f379f802516e","Type":"ContainerDied","Data":"ce9239037d2e5034b8f4ac88a4a0fd9fc069186f12c4253005969ac049f7c1d8"}
Aug 04 18:16:11.809963 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:11.809948    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"
Aug 04 18:16:11.810455 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:11.809994    2543 scope.go:117] "RemoveContainer" containerID="db96fc5357dbaafbaab1d37fc507c0244ef9503cb9e5e2e8c1efa7b57b0aa888"
Aug 04 18:16:11.817962 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:11.817946    2543 scope.go:117] "RemoveContainer" containerID="a21d0951b1be44a150db3b6492a9ffdf02083f9126b47078cc077e3c15ecb5cc"
Aug 04 18:16:11.825079 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:11.825063    2543 scope.go:117] "RemoveContainer" containerID="4c3b1dfd61a992e543dd5012f2ab8583f01bdad84b85ab072665517aca485ff7"
Aug 04 18:16:11.832950 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:11.832930    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"]
Aug 04 18:16:11.836941 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:11.836921    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-99f90-predictor-6844b5d67f-6hpmv"]
Aug 04 18:16:13.574220 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:13.574180    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" path="/var/lib/kubelet/pods/d7355f5c-78a5-4cab-b175-f379f802516e/volumes"
Aug 04 18:16:26.586105 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586070    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"]
Aug 04 18:16:26.586496 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586425    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="storage-initializer"
Aug 04 18:16:26.586496 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586439    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="storage-initializer"
Aug 04 18:16:26.586496 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586448    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kserve-container"
Aug 04 18:16:26.586496 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586453    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kserve-container"
Aug 04 18:16:26.586496 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586467    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="storage-initializer"
Aug 04 18:16:26.586496 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586473    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="storage-initializer"
Aug 04 18:16:26.586496 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586480    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="storage-initializer"
Aug 04 18:16:26.586496 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586486    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="storage-initializer"
Aug 04 18:16:26.586496 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586494    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kube-rbac-proxy"
Aug 04 18:16:26.586496 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586499    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kube-rbac-proxy"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586507    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kube-rbac-proxy"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586513    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kube-rbac-proxy"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586519    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kube-rbac-proxy"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586524    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kube-rbac-proxy"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586532    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kserve-container"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586538    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kserve-container"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586545    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="storage-initializer"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586550    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="storage-initializer"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586557    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="kserve-container"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586563    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="kserve-container"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586570    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="kube-rbac-proxy"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586575    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="kube-rbac-proxy"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586580    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kserve-container"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586585    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kserve-container"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586630    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="kube-rbac-proxy"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586639    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kserve-container"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586647    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kserve-container"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586653    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7355f5c-78a5-4cab-b175-f379f802516e" containerName="kube-rbac-proxy"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586659    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kube-rbac-proxy"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586665    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e74e25e-c752-4fcd-83d3-3c7dc26df936" containerName="kserve-container"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586672    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="52069c82-27a0-4aeb-9be5-63288ef62e3b" containerName="kserve-container"
Aug 04 18:16:26.586792 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.586679    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="b09ebb2d-7f40-4ebd-a850-c58d32199f96" containerName="kube-rbac-proxy"
Aug 04 18:16:26.589819 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.589801    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.592666 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.592647    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-fe589-predictor-serving-cert\""
Aug 04 18:16:26.592777 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.592653    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:16:26.593850 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.593829    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vsp4f\""
Aug 04 18:16:26.594005 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.593869    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:16:26.594005 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.593929    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:16:26.594005 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.593952    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-fe589-kube-rbac-proxy-sar-config\""
Aug 04 18:16:26.594005 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.593972    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:16:26.601638 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.601617    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"]
Aug 04 18:16:26.721956 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.721925    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4296930-5b2f-4faa-a85b-76dc6bab2697-cabundle-cert\") pod \"isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.721956 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.721957    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4296930-5b2f-4faa-a85b-76dc6bab2697-kserve-provision-location\") pod \"isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.722140 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.721994    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4296930-5b2f-4faa-a85b-76dc6bab2697-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.722140 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.722072    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4296930-5b2f-4faa-a85b-76dc6bab2697-proxy-tls\") pod \"isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.722140 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.722105    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swzgc\" (UniqueName: \"kubernetes.io/projected/c4296930-5b2f-4faa-a85b-76dc6bab2697-kube-api-access-swzgc\") pod \"isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.823198 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.823173    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4296930-5b2f-4faa-a85b-76dc6bab2697-proxy-tls\") pod \"isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.823296 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.823205    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swzgc\" (UniqueName: \"kubernetes.io/projected/c4296930-5b2f-4faa-a85b-76dc6bab2697-kube-api-access-swzgc\") pod \"isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.823296 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.823256    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4296930-5b2f-4faa-a85b-76dc6bab2697-cabundle-cert\") pod \"isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.823296 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.823281    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4296930-5b2f-4faa-a85b-76dc6bab2697-kserve-provision-location\") pod \"isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.823432 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.823357    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4296930-5b2f-4faa-a85b-76dc6bab2697-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.823820 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.823797    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4296930-5b2f-4faa-a85b-76dc6bab2697-kserve-provision-location\") pod \"isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.824004 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.823987    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4296930-5b2f-4faa-a85b-76dc6bab2697-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.824086 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.824007    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4296930-5b2f-4faa-a85b-76dc6bab2697-cabundle-cert\") pod \"isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.825703 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.825684    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4296930-5b2f-4faa-a85b-76dc6bab2697-proxy-tls\") pod \"isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.832485 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.832458    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swzgc\" (UniqueName: \"kubernetes.io/projected/c4296930-5b2f-4faa-a85b-76dc6bab2697-kube-api-access-swzgc\") pod \"isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:26.899536 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:26.899512    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:27.033347 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:27.033323    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"]
Aug 04 18:16:27.034796 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:16:27.034757    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4296930_5b2f_4faa_a85b_76dc6bab2697.slice/crio-bebbf156ec58b78468018d0447f9f62a59873f05bab3e93e1909a2d49621b972 WatchSource:0}: Error finding container bebbf156ec58b78468018d0447f9f62a59873f05bab3e93e1909a2d49621b972: Status 404 returned error can't find the container with id bebbf156ec58b78468018d0447f9f62a59873f05bab3e93e1909a2d49621b972
Aug 04 18:16:27.871872 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:27.871833    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" event={"ID":"c4296930-5b2f-4faa-a85b-76dc6bab2697","Type":"ContainerStarted","Data":"49e5d7eb5dc1b33f41f20b0dde028f5229079fd7f73b12b39c78aab8ab8cc133"}
Aug 04 18:16:27.872261 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:27.871902    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" event={"ID":"c4296930-5b2f-4faa-a85b-76dc6bab2697","Type":"ContainerStarted","Data":"bebbf156ec58b78468018d0447f9f62a59873f05bab3e93e1909a2d49621b972"}
Aug 04 18:16:31.889779 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:31.889749    2543 generic.go:356] "Generic (PLEG): container finished" podID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerID="49e5d7eb5dc1b33f41f20b0dde028f5229079fd7f73b12b39c78aab8ab8cc133" exitCode=0
Aug 04 18:16:31.890264 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:31.889829    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" event={"ID":"c4296930-5b2f-4faa-a85b-76dc6bab2697","Type":"ContainerDied","Data":"49e5d7eb5dc1b33f41f20b0dde028f5229079fd7f73b12b39c78aab8ab8cc133"}
Aug 04 18:16:32.896656 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:32.896623    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" event={"ID":"c4296930-5b2f-4faa-a85b-76dc6bab2697","Type":"ContainerStarted","Data":"6b881ed4be8ecdc752051c39092bd1221c3811ec8d6901f5c3d3e063146fbfb2"}
Aug 04 18:16:32.897058 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:32.896686    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" event={"ID":"c4296930-5b2f-4faa-a85b-76dc6bab2697","Type":"ContainerStarted","Data":"d565a94ffc1e0a701aea60d33ea00d72750d405db9e733da32cac30acedb2e4f"}
Aug 04 18:16:32.897058 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:32.896713    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" event={"ID":"c4296930-5b2f-4faa-a85b-76dc6bab2697","Type":"ContainerStarted","Data":"7685c7000600cb0678c7b36cd0602fe668249864af79a9d6d0f01b6fc0e9f209"}
Aug 04 18:16:32.897058 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:32.897009    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:32.897193 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:32.897177    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:32.898602 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:32.898576    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:16:32.921038 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:32.920995    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podStartSLOduration=3.106534167 podStartE2EDuration="6.920983845s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.814449678s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:16:32.918446288 +0000 UTC m=+995.889090651" watchObservedRunningTime="2026-08-04 18:16:32.920983845 +0000 UTC m=+995.891628185"
Aug 04 18:16:33.900730 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:33.900697    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:33.901154 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:33.900913    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:16:33.902071 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:33.902036    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:16:34.904772 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:34.904719    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:16:34.905190 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:34.905129    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:16:39.908679 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:39.908649    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:16:39.909240 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:39.909208    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:16:39.909538 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:39.909511    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:16:49.909502 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:49.909458    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:16:49.909888 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:49.909839    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:16:59.909875 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:59.909817    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:16:59.910280 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:16:59.910239    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:17:09.909120 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:09.909077    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:17:09.909546 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:09.909518    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:17:19.909120 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:19.909075    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:17:19.909628 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:19.909600    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:17:29.909685 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:29.909629    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:17:29.910165 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:29.910135    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:17:39.909846 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:39.909807    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:17:39.910363 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:39.910214    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:17:42.570107 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:42.570074    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:17:42.570559 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:42.570390    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:17:51.727428 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:51.727356    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"]
Aug 04 18:17:51.727954 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:51.727609    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container" containerID="cri-o://7685c7000600cb0678c7b36cd0602fe668249864af79a9d6d0f01b6fc0e9f209" gracePeriod=30
Aug 04 18:17:51.727954 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:51.727621    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent" containerID="cri-o://6b881ed4be8ecdc752051c39092bd1221c3811ec8d6901f5c3d3e063146fbfb2" gracePeriod=30
Aug 04 18:17:51.727954 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:51.727657    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kube-rbac-proxy" containerID="cri-o://d565a94ffc1e0a701aea60d33ea00d72750d405db9e733da32cac30acedb2e4f" gracePeriod=30
Aug 04 18:17:51.796323 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:51.796282    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"]
Aug 04 18:17:51.799795 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:51.799776    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:51.802303 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:51.802282    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-a9b62-kube-rbac-proxy-sar-config\""
Aug 04 18:17:51.802407 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:51.802283    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-a9b62-predictor-serving-cert\""
Aug 04 18:17:51.810708 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:51.810689    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"]
Aug 04 18:17:51.960477 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:51.960451    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:51.960577 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:51.960486    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b32d8a5-1a56-406a-8a30-5bde553afe4f-proxy-tls\") pod \"isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:51.960577 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:51.960529    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:51.960577 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:51.960553    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b32d8a5-1a56-406a-8a30-5bde553afe4f-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:51.960687 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:51.960591    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7cl4\" (UniqueName: \"kubernetes.io/projected/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kube-api-access-b7cl4\") pod \"isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:52.061834 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.061772    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7cl4\" (UniqueName: \"kubernetes.io/projected/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kube-api-access-b7cl4\") pod \"isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:52.061918 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.061833    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:52.061918 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.061871    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b32d8a5-1a56-406a-8a30-5bde553afe4f-proxy-tls\") pod \"isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:52.062004 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.061929    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:52.062004 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.061952    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b32d8a5-1a56-406a-8a30-5bde553afe4f-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:52.062264 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.062240    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:52.062630 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.062607    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:52.062706 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.062614    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b32d8a5-1a56-406a-8a30-5bde553afe4f-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:52.064246 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.064225    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b32d8a5-1a56-406a-8a30-5bde553afe4f-proxy-tls\") pod \"isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:52.070904 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.070882    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7cl4\" (UniqueName: \"kubernetes.io/projected/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kube-api-access-b7cl4\") pod \"isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:52.109838 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.109816    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:52.219189 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.219162    2543 generic.go:356] "Generic (PLEG): container finished" podID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerID="d565a94ffc1e0a701aea60d33ea00d72750d405db9e733da32cac30acedb2e4f" exitCode=2
Aug 04 18:17:52.219328 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.219231    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" event={"ID":"c4296930-5b2f-4faa-a85b-76dc6bab2697","Type":"ContainerDied","Data":"d565a94ffc1e0a701aea60d33ea00d72750d405db9e733da32cac30acedb2e4f"}
Aug 04 18:17:52.244875 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.244854    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"]
Aug 04 18:17:52.247765 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:17:52.247739    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b32d8a5_1a56_406a_8a30_5bde553afe4f.slice/crio-26cee1128729e5b2a320f0851e649326f61408ab4240219381c99b0ccb78c681 WatchSource:0}: Error finding container 26cee1128729e5b2a320f0851e649326f61408ab4240219381c99b0ccb78c681: Status 404 returned error can't find the container with id 26cee1128729e5b2a320f0851e649326f61408ab4240219381c99b0ccb78c681
Aug 04 18:17:52.569835 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.569797    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:17:52.570168 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:52.570140    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:17:53.224481 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:53.224447    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" event={"ID":"7b32d8a5-1a56-406a-8a30-5bde553afe4f","Type":"ContainerStarted","Data":"57e015c6bd364cc482362e3618c7e281020ac25b0776a24f8fdebfa244f2aac3"}
Aug 04 18:17:53.224931 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:53.224516    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" event={"ID":"7b32d8a5-1a56-406a-8a30-5bde553afe4f","Type":"ContainerStarted","Data":"26cee1128729e5b2a320f0851e649326f61408ab4240219381c99b0ccb78c681"}
Aug 04 18:17:54.905717 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:54.905681    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Aug 04 18:17:56.238435 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:56.238362    2543 generic.go:356] "Generic (PLEG): container finished" podID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerID="57e015c6bd364cc482362e3618c7e281020ac25b0776a24f8fdebfa244f2aac3" exitCode=0
Aug 04 18:17:56.238768 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:56.238434    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" event={"ID":"7b32d8a5-1a56-406a-8a30-5bde553afe4f","Type":"ContainerDied","Data":"57e015c6bd364cc482362e3618c7e281020ac25b0776a24f8fdebfa244f2aac3"}
Aug 04 18:17:56.241149 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:56.241128    2543 generic.go:356] "Generic (PLEG): container finished" podID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerID="7685c7000600cb0678c7b36cd0602fe668249864af79a9d6d0f01b6fc0e9f209" exitCode=0
Aug 04 18:17:56.241222 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:56.241197    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" event={"ID":"c4296930-5b2f-4faa-a85b-76dc6bab2697","Type":"ContainerDied","Data":"7685c7000600cb0678c7b36cd0602fe668249864af79a9d6d0f01b6fc0e9f209"}
Aug 04 18:17:57.247975 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:57.247936    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" event={"ID":"7b32d8a5-1a56-406a-8a30-5bde553afe4f","Type":"ContainerStarted","Data":"617388a36568c684e5d4e60e66c493f5a5310be9778245c5c0e38b8bf3e1b36d"}
Aug 04 18:17:57.248498 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:57.248024    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" event={"ID":"7b32d8a5-1a56-406a-8a30-5bde553afe4f","Type":"ContainerStarted","Data":"962b8de03e9c33df8d8458657095a896692680c148350454759c113cc882c369"}
Aug 04 18:17:57.248498 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:57.248264    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:57.273573 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:57.273529    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podStartSLOduration=2.626762545 podStartE2EDuration="6.27351665s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.646754105s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:57.270991983 +0000 UTC m=+1080.241636323" watchObservedRunningTime="2026-08-04 18:17:57.27351665 +0000 UTC m=+1080.244160990"
Aug 04 18:17:58.251745 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:58.251710    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:17:58.252520 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:58.252496    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:17:59.257112 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:59.257061    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:17:59.905189 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:17:59.905151    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Aug 04 18:18:02.569876 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:02.569830    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:18:02.570411 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:02.570231    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:18:04.261822 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:04.261788    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:18:04.262472 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:04.262446    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:18:04.905441 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:04.905408    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Aug 04 18:18:04.905598 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:04.905508    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:18:09.905169 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:09.905123    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Aug 04 18:18:12.570399 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:12.570349    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:18:12.570876 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:12.570543    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:18:12.570876 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:12.570656    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:18:12.570876 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:12.570792    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:18:14.262605 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:14.262561    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:18:14.905491 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:14.905457    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Aug 04 18:18:19.905787 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:19.905737    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Aug 04 18:18:22.353830 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.353799    2543 generic.go:356] "Generic (PLEG): container finished" podID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerID="6b881ed4be8ecdc752051c39092bd1221c3811ec8d6901f5c3d3e063146fbfb2" exitCode=137
Aug 04 18:18:22.354244 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.353865    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" event={"ID":"c4296930-5b2f-4faa-a85b-76dc6bab2697","Type":"ContainerDied","Data":"6b881ed4be8ecdc752051c39092bd1221c3811ec8d6901f5c3d3e063146fbfb2"}
Aug 04 18:18:22.376295 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.376272    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:18:22.483156 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.483132    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4296930-5b2f-4faa-a85b-76dc6bab2697-kserve-provision-location\") pod \"c4296930-5b2f-4faa-a85b-76dc6bab2697\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") "
Aug 04 18:18:22.483260 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.483174    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4296930-5b2f-4faa-a85b-76dc6bab2697-kube-rbac-proxy-sar-config\") pod \"c4296930-5b2f-4faa-a85b-76dc6bab2697\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") "
Aug 04 18:18:22.483260 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.483204    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4296930-5b2f-4faa-a85b-76dc6bab2697-proxy-tls\") pod \"c4296930-5b2f-4faa-a85b-76dc6bab2697\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") "
Aug 04 18:18:22.483260 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.483253    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4296930-5b2f-4faa-a85b-76dc6bab2697-cabundle-cert\") pod \"c4296930-5b2f-4faa-a85b-76dc6bab2697\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") "
Aug 04 18:18:22.483427 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.483293    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swzgc\" (UniqueName: \"kubernetes.io/projected/c4296930-5b2f-4faa-a85b-76dc6bab2697-kube-api-access-swzgc\") pod \"c4296930-5b2f-4faa-a85b-76dc6bab2697\" (UID: \"c4296930-5b2f-4faa-a85b-76dc6bab2697\") "
Aug 04 18:18:22.483490 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.483397    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4296930-5b2f-4faa-a85b-76dc6bab2697-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c4296930-5b2f-4faa-a85b-76dc6bab2697" (UID: "c4296930-5b2f-4faa-a85b-76dc6bab2697"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:18:22.483591 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.483567    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4296930-5b2f-4faa-a85b-76dc6bab2697-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:18:22.483708 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.483608    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4296930-5b2f-4faa-a85b-76dc6bab2697-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c4296930-5b2f-4faa-a85b-76dc6bab2697" (UID: "c4296930-5b2f-4faa-a85b-76dc6bab2697"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:18:22.483708 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.483682    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4296930-5b2f-4faa-a85b-76dc6bab2697-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c4296930-5b2f-4faa-a85b-76dc6bab2697" (UID: "c4296930-5b2f-4faa-a85b-76dc6bab2697"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:18:22.485345 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.485302    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4296930-5b2f-4faa-a85b-76dc6bab2697-kube-api-access-swzgc" (OuterVolumeSpecName: "kube-api-access-swzgc") pod "c4296930-5b2f-4faa-a85b-76dc6bab2697" (UID: "c4296930-5b2f-4faa-a85b-76dc6bab2697"). InnerVolumeSpecName "kube-api-access-swzgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:18:22.485428 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.485349    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4296930-5b2f-4faa-a85b-76dc6bab2697-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c4296930-5b2f-4faa-a85b-76dc6bab2697" (UID: "c4296930-5b2f-4faa-a85b-76dc6bab2697"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:18:22.584859 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.584806    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swzgc\" (UniqueName: \"kubernetes.io/projected/c4296930-5b2f-4faa-a85b-76dc6bab2697-kube-api-access-swzgc\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:18:22.584859 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.584827    2543 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4296930-5b2f-4faa-a85b-76dc6bab2697-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:18:22.584859 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.584837    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4296930-5b2f-4faa-a85b-76dc6bab2697-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:18:22.584859 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:22.584846    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4296930-5b2f-4faa-a85b-76dc6bab2697-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:18:23.359974 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:23.359932    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7" event={"ID":"c4296930-5b2f-4faa-a85b-76dc6bab2697","Type":"ContainerDied","Data":"bebbf156ec58b78468018d0447f9f62a59873f05bab3e93e1909a2d49621b972"}
Aug 04 18:18:23.360559 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:23.359985    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"
Aug 04 18:18:23.360559 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:23.360032    2543 scope.go:117] "RemoveContainer" containerID="6b881ed4be8ecdc752051c39092bd1221c3811ec8d6901f5c3d3e063146fbfb2"
Aug 04 18:18:23.368411 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:23.368393    2543 scope.go:117] "RemoveContainer" containerID="d565a94ffc1e0a701aea60d33ea00d72750d405db9e733da32cac30acedb2e4f"
Aug 04 18:18:23.375958 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:23.375943    2543 scope.go:117] "RemoveContainer" containerID="7685c7000600cb0678c7b36cd0602fe668249864af79a9d6d0f01b6fc0e9f209"
Aug 04 18:18:23.383473 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:23.383453    2543 scope.go:117] "RemoveContainer" containerID="49e5d7eb5dc1b33f41f20b0dde028f5229079fd7f73b12b39c78aab8ab8cc133"
Aug 04 18:18:23.384504 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:23.384483    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"]
Aug 04 18:18:23.389074 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:23.389053    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-fe589-predictor-9675d5fc7-bfsm7"]
Aug 04 18:18:23.574114 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:23.574092    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" path="/var/lib/kubelet/pods/c4296930-5b2f-4faa-a85b-76dc6bab2697/volumes"
Aug 04 18:18:24.262526 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:24.262481    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:18:34.262899 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:34.262858    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:18:44.263027 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:44.262986    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:18:54.263153 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:18:54.263118    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:19:04.263239 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:19:04.263204    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:19:14.262487 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:19:14.262445    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:19:24.263185 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:19:24.263153    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:19:33.570125 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:19:33.570085    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:19:43.570597 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:19:43.570551    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:19:53.570506 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:19:53.570462    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:19:57.527547 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:19:57.527509    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/1.log"
Aug 04 18:19:57.527985 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:19:57.527840    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/1.log"
Aug 04 18:20:03.570866 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:03.570827    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:20:13.570003 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:13.569963    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:20:23.574472 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:23.574447    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:20:31.944853 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:31.944823    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"]
Aug 04 18:20:31.945278 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:31.945046    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" containerID="cri-o://962b8de03e9c33df8d8458657095a896692680c148350454759c113cc882c369" gracePeriod=30
Aug 04 18:20:31.945278 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:31.945095    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kube-rbac-proxy" containerID="cri-o://617388a36568c684e5d4e60e66c493f5a5310be9778245c5c0e38b8bf3e1b36d" gracePeriod=30
Aug 04 18:20:32.867233 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:32.867204    2543 generic.go:356] "Generic (PLEG): container finished" podID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerID="617388a36568c684e5d4e60e66c493f5a5310be9778245c5c0e38b8bf3e1b36d" exitCode=2
Aug 04 18:20:32.867396 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:32.867272    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" event={"ID":"7b32d8a5-1a56-406a-8a30-5bde553afe4f","Type":"ContainerDied","Data":"617388a36568c684e5d4e60e66c493f5a5310be9778245c5c0e38b8bf3e1b36d"}
Aug 04 18:20:33.570013 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:33.569977    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:20:34.257859 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:34.257826    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Aug 04 18:20:37.164766 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.164734    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"]
Aug 04 18:20:37.165128 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.165057    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container"
Aug 04 18:20:37.165128 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.165069    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container"
Aug 04 18:20:37.165128 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.165078    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="storage-initializer"
Aug 04 18:20:37.165128 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.165083    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="storage-initializer"
Aug 04 18:20:37.165128 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.165090    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kube-rbac-proxy"
Aug 04 18:20:37.165128 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.165096    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kube-rbac-proxy"
Aug 04 18:20:37.165128 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.165107    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent"
Aug 04 18:20:37.165128 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.165115    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent"
Aug 04 18:20:37.165426 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.165166    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kserve-container"
Aug 04 18:20:37.165426 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.165175    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="kube-rbac-proxy"
Aug 04 18:20:37.165426 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.165182    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4296930-5b2f-4faa-a85b-76dc6bab2697" containerName="agent"
Aug 04 18:20:37.168373 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.168353    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.170926 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.170906    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Aug 04 18:20:37.171000 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.170912    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Aug 04 18:20:37.176155 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.176136    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"]
Aug 04 18:20:37.179420 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.179403    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.181117 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.181097    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"]
Aug 04 18:20:37.181841 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.181825    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Aug 04 18:20:37.196238 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.196216    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"]
Aug 04 18:20:37.246412 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.246391    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qgmv\" (UniqueName: \"kubernetes.io/projected/37d12933-dfdc-410a-a449-b016ed34a8af-kube-api-access-8qgmv\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.246520 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.246432    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37d12933-dfdc-410a-a449-b016ed34a8af-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.246520 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.246464    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d12933-dfdc-410a-a449-b016ed34a8af-proxy-tls\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.246600 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.246554    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.246600 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.246581    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kserve-provision-location\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.246699 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.246607    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ea85fc4-2d8b-43f8-90e2-e380068b629c-cabundle-cert\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.246699 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.246675    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea85fc4-2d8b-43f8-90e2-e380068b629c-proxy-tls\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.246801 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.246700    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37d12933-dfdc-410a-a449-b016ed34a8af-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.246801 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.246730    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37d12933-dfdc-410a-a449-b016ed34a8af-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.246801 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.246760    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6q4k\" (UniqueName: \"kubernetes.io/projected/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kube-api-access-q6q4k\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.347944 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.347909    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.347944 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.347947    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kserve-provision-location\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.348121 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.347974    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ea85fc4-2d8b-43f8-90e2-e380068b629c-cabundle-cert\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.348121 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.348010    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea85fc4-2d8b-43f8-90e2-e380068b629c-proxy-tls\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.348221 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.348113    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37d12933-dfdc-410a-a449-b016ed34a8af-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.348221 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.348155    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37d12933-dfdc-410a-a449-b016ed34a8af-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.348359 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:20:37.348259    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Aug 04 18:20:37.348359 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:20:37.348345    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea85fc4-2d8b-43f8-90e2-e380068b629c-proxy-tls podName:6ea85fc4-2d8b-43f8-90e2-e380068b629c nodeName:}" failed. No retries permitted until 2026-08-04 18:20:37.848302293 +0000 UTC m=+1240.818946616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6ea85fc4-2d8b-43f8-90e2-e380068b629c-proxy-tls") pod "isvc-canary-create-predictor-65d8dbfcdb-dpg7m" (UID: "6ea85fc4-2d8b-43f8-90e2-e380068b629c") : secret "isvc-canary-create-predictor-serving-cert" not found
Aug 04 18:20:37.348482 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.348412    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kserve-provision-location\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.348482 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.348258    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6q4k\" (UniqueName: \"kubernetes.io/projected/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kube-api-access-q6q4k\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.348586 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.348483    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qgmv\" (UniqueName: \"kubernetes.io/projected/37d12933-dfdc-410a-a449-b016ed34a8af-kube-api-access-8qgmv\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.348586 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.348519    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37d12933-dfdc-410a-a449-b016ed34a8af-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.348586 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.348552    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d12933-dfdc-410a-a449-b016ed34a8af-proxy-tls\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.348733 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.348638    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37d12933-dfdc-410a-a449-b016ed34a8af-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.348733 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:20:37.348681    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Aug 04 18:20:37.348818 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:20:37.348740    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d12933-dfdc-410a-a449-b016ed34a8af-proxy-tls podName:37d12933-dfdc-410a-a449-b016ed34a8af nodeName:}" failed. No retries permitted until 2026-08-04 18:20:37.848722279 +0000 UTC m=+1240.819366616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/37d12933-dfdc-410a-a449-b016ed34a8af-proxy-tls") pod "isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" (UID: "37d12933-dfdc-410a-a449-b016ed34a8af") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Aug 04 18:20:37.348818 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.348776    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ea85fc4-2d8b-43f8-90e2-e380068b629c-cabundle-cert\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.348818 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.348794    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.348923 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.348881    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37d12933-dfdc-410a-a449-b016ed34a8af-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.349039 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.349023    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37d12933-dfdc-410a-a449-b016ed34a8af-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.359713 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.359687    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qgmv\" (UniqueName: \"kubernetes.io/projected/37d12933-dfdc-410a-a449-b016ed34a8af-kube-api-access-8qgmv\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.359889 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.359873    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6q4k\" (UniqueName: \"kubernetes.io/projected/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kube-api-access-q6q4k\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.851990 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.851959    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea85fc4-2d8b-43f8-90e2-e380068b629c-proxy-tls\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:37.852142 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.852021    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d12933-dfdc-410a-a449-b016ed34a8af-proxy-tls\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.854299 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.854274    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d12933-dfdc-410a-a449-b016ed34a8af-proxy-tls\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-2bmm8\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:37.854422 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:37.854325    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea85fc4-2d8b-43f8-90e2-e380068b629c-proxy-tls\") pod \"isvc-canary-create-predictor-65d8dbfcdb-dpg7m\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:38.078133 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:38.078100    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:38.089454 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:38.089433    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:38.221901 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:38.221836    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"]
Aug 04 18:20:38.224083 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:20:38.224052    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea85fc4_2d8b_43f8_90e2_e380068b629c.slice/crio-39c656cebbc4714b5fa2a66fa9de07a872f4ddeda0959af093c27e0f28846de0 WatchSource:0}: Error finding container 39c656cebbc4714b5fa2a66fa9de07a872f4ddeda0959af093c27e0f28846de0: Status 404 returned error can't find the container with id 39c656cebbc4714b5fa2a66fa9de07a872f4ddeda0959af093c27e0f28846de0
Aug 04 18:20:38.236757 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:38.236734    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"]
Aug 04 18:20:38.239563 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:20:38.239542    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d12933_dfdc_410a_a449_b016ed34a8af.slice/crio-416a29ef439d753b1dd8ffdc6a90d00173f98d9d0c180f722cbd6bab52d3c791 WatchSource:0}: Error finding container 416a29ef439d753b1dd8ffdc6a90d00173f98d9d0c180f722cbd6bab52d3c791: Status 404 returned error can't find the container with id 416a29ef439d753b1dd8ffdc6a90d00173f98d9d0c180f722cbd6bab52d3c791
Aug 04 18:20:38.890366 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:38.890324    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" event={"ID":"6ea85fc4-2d8b-43f8-90e2-e380068b629c","Type":"ContainerStarted","Data":"def66af1643bd60fa2f8d60aaae2e49aeffe56300b25395678f8e08768c48bce"}
Aug 04 18:20:38.890546 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:38.890407    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" event={"ID":"6ea85fc4-2d8b-43f8-90e2-e380068b629c","Type":"ContainerStarted","Data":"39c656cebbc4714b5fa2a66fa9de07a872f4ddeda0959af093c27e0f28846de0"}
Aug 04 18:20:38.891749 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:38.891721    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" event={"ID":"37d12933-dfdc-410a-a449-b016ed34a8af","Type":"ContainerStarted","Data":"1deedc210307d67c24bacb403a9b29ababb76153b1bcedd5740a9703e509b17c"}
Aug 04 18:20:38.891873 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:38.891779    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" event={"ID":"37d12933-dfdc-410a-a449-b016ed34a8af","Type":"ContainerStarted","Data":"416a29ef439d753b1dd8ffdc6a90d00173f98d9d0c180f722cbd6bab52d3c791"}
Aug 04 18:20:39.258067 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.257991    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Aug 04 18:20:39.689703 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.689673    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:20:39.768830 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.768804    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b32d8a5-1a56-406a-8a30-5bde553afe4f-cabundle-cert\") pod \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") "
Aug 04 18:20:39.768941 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.768834    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7cl4\" (UniqueName: \"kubernetes.io/projected/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kube-api-access-b7cl4\") pod \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") "
Aug 04 18:20:39.768941 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.768866    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b32d8a5-1a56-406a-8a30-5bde553afe4f-proxy-tls\") pod \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") "
Aug 04 18:20:39.768941 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.768925    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kserve-provision-location\") pod \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") "
Aug 04 18:20:39.769045 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.768970    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kube-rbac-proxy-sar-config\") pod \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\" (UID: \"7b32d8a5-1a56-406a-8a30-5bde553afe4f\") "
Aug 04 18:20:39.769280 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.769239    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b32d8a5-1a56-406a-8a30-5bde553afe4f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7b32d8a5-1a56-406a-8a30-5bde553afe4f" (UID: "7b32d8a5-1a56-406a-8a30-5bde553afe4f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:20:39.769280 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.769263    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b32d8a5-1a56-406a-8a30-5bde553afe4f" (UID: "7b32d8a5-1a56-406a-8a30-5bde553afe4f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:20:39.769467 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.769448    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b32d8a5-1a56-406a-8a30-5bde553afe4f" (UID: "7b32d8a5-1a56-406a-8a30-5bde553afe4f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:20:39.770951 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.770932    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b32d8a5-1a56-406a-8a30-5bde553afe4f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b32d8a5-1a56-406a-8a30-5bde553afe4f" (UID: "7b32d8a5-1a56-406a-8a30-5bde553afe4f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:20:39.771006 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.770966    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kube-api-access-b7cl4" (OuterVolumeSpecName: "kube-api-access-b7cl4") pod "7b32d8a5-1a56-406a-8a30-5bde553afe4f" (UID: "7b32d8a5-1a56-406a-8a30-5bde553afe4f"). InnerVolumeSpecName "kube-api-access-b7cl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:20:39.870043 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.870018    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b32d8a5-1a56-406a-8a30-5bde553afe4f-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:20:39.870043 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.870040    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:20:39.870188 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.870053    2543 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:20:39.870188 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.870066    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b32d8a5-1a56-406a-8a30-5bde553afe4f-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:20:39.870188 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.870078    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7cl4\" (UniqueName: \"kubernetes.io/projected/7b32d8a5-1a56-406a-8a30-5bde553afe4f-kube-api-access-b7cl4\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:20:39.897368 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.897345    2543 generic.go:356] "Generic (PLEG): container finished" podID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerID="962b8de03e9c33df8d8458657095a896692680c148350454759c113cc882c369" exitCode=0
Aug 04 18:20:39.897459 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.897402    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"
Aug 04 18:20:39.897459 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.897430    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" event={"ID":"7b32d8a5-1a56-406a-8a30-5bde553afe4f","Type":"ContainerDied","Data":"962b8de03e9c33df8d8458657095a896692680c148350454759c113cc882c369"}
Aug 04 18:20:39.897543 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.897490    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2" event={"ID":"7b32d8a5-1a56-406a-8a30-5bde553afe4f","Type":"ContainerDied","Data":"26cee1128729e5b2a320f0851e649326f61408ab4240219381c99b0ccb78c681"}
Aug 04 18:20:39.897543 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.897516    2543 scope.go:117] "RemoveContainer" containerID="617388a36568c684e5d4e60e66c493f5a5310be9778245c5c0e38b8bf3e1b36d"
Aug 04 18:20:39.905681 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.905665    2543 scope.go:117] "RemoveContainer" containerID="962b8de03e9c33df8d8458657095a896692680c148350454759c113cc882c369"
Aug 04 18:20:39.912706 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.912689    2543 scope.go:117] "RemoveContainer" containerID="57e015c6bd364cc482362e3618c7e281020ac25b0776a24f8fdebfa244f2aac3"
Aug 04 18:20:39.919581 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.919563    2543 scope.go:117] "RemoveContainer" containerID="617388a36568c684e5d4e60e66c493f5a5310be9778245c5c0e38b8bf3e1b36d"
Aug 04 18:20:39.919854 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:20:39.919836    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"617388a36568c684e5d4e60e66c493f5a5310be9778245c5c0e38b8bf3e1b36d\": container with ID starting with 617388a36568c684e5d4e60e66c493f5a5310be9778245c5c0e38b8bf3e1b36d not found: ID does not exist" containerID="617388a36568c684e5d4e60e66c493f5a5310be9778245c5c0e38b8bf3e1b36d"
Aug 04 18:20:39.919902 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.919863    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"617388a36568c684e5d4e60e66c493f5a5310be9778245c5c0e38b8bf3e1b36d"} err="failed to get container status \"617388a36568c684e5d4e60e66c493f5a5310be9778245c5c0e38b8bf3e1b36d\": rpc error: code = NotFound desc = could not find container \"617388a36568c684e5d4e60e66c493f5a5310be9778245c5c0e38b8bf3e1b36d\": container with ID starting with 617388a36568c684e5d4e60e66c493f5a5310be9778245c5c0e38b8bf3e1b36d not found: ID does not exist"
Aug 04 18:20:39.919902 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.919880    2543 scope.go:117] "RemoveContainer" containerID="962b8de03e9c33df8d8458657095a896692680c148350454759c113cc882c369"
Aug 04 18:20:39.919997 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.919921    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"]
Aug 04 18:20:39.920161 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:20:39.920142    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962b8de03e9c33df8d8458657095a896692680c148350454759c113cc882c369\": container with ID starting with 962b8de03e9c33df8d8458657095a896692680c148350454759c113cc882c369 not found: ID does not exist" containerID="962b8de03e9c33df8d8458657095a896692680c148350454759c113cc882c369"
Aug 04 18:20:39.920208 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.920166    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962b8de03e9c33df8d8458657095a896692680c148350454759c113cc882c369"} err="failed to get container status \"962b8de03e9c33df8d8458657095a896692680c148350454759c113cc882c369\": rpc error: code = NotFound desc = could not find container \"962b8de03e9c33df8d8458657095a896692680c148350454759c113cc882c369\": container with ID starting with 962b8de03e9c33df8d8458657095a896692680c148350454759c113cc882c369 not found: ID does not exist"
Aug 04 18:20:39.920208 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.920181    2543 scope.go:117] "RemoveContainer" containerID="57e015c6bd364cc482362e3618c7e281020ac25b0776a24f8fdebfa244f2aac3"
Aug 04 18:20:39.920405 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:20:39.920390    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e015c6bd364cc482362e3618c7e281020ac25b0776a24f8fdebfa244f2aac3\": container with ID starting with 57e015c6bd364cc482362e3618c7e281020ac25b0776a24f8fdebfa244f2aac3 not found: ID does not exist" containerID="57e015c6bd364cc482362e3618c7e281020ac25b0776a24f8fdebfa244f2aac3"
Aug 04 18:20:39.920451 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.920410    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e015c6bd364cc482362e3618c7e281020ac25b0776a24f8fdebfa244f2aac3"} err="failed to get container status \"57e015c6bd364cc482362e3618c7e281020ac25b0776a24f8fdebfa244f2aac3\": rpc error: code = NotFound desc = could not find container \"57e015c6bd364cc482362e3618c7e281020ac25b0776a24f8fdebfa244f2aac3\": container with ID starting with 57e015c6bd364cc482362e3618c7e281020ac25b0776a24f8fdebfa244f2aac3 not found: ID does not exist"
Aug 04 18:20:39.923949 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:39.923932    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9b62-predictor-7856bb6d69-tgdg2"]
Aug 04 18:20:41.574609 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:41.574580    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" path="/var/lib/kubelet/pods/7b32d8a5-1a56-406a-8a30-5bde553afe4f/volumes"
Aug 04 18:20:41.907497 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:41.907457    2543 generic.go:356] "Generic (PLEG): container finished" podID="37d12933-dfdc-410a-a449-b016ed34a8af" containerID="1deedc210307d67c24bacb403a9b29ababb76153b1bcedd5740a9703e509b17c" exitCode=0
Aug 04 18:20:41.907655 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:41.907515    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" event={"ID":"37d12933-dfdc-410a-a449-b016ed34a8af","Type":"ContainerDied","Data":"1deedc210307d67c24bacb403a9b29ababb76153b1bcedd5740a9703e509b17c"}
Aug 04 18:20:42.914201 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:42.914167    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" event={"ID":"37d12933-dfdc-410a-a449-b016ed34a8af","Type":"ContainerStarted","Data":"f4651340f3b816ee0e099aaf5beeb060b2107f1264197eb2659b2b8c757a099e"}
Aug 04 18:20:42.914633 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:42.914236    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" event={"ID":"37d12933-dfdc-410a-a449-b016ed34a8af","Type":"ContainerStarted","Data":"c8a248c271618d328cb1be397e1924736d1ea434a69871da466d64b8b5f15a58"}
Aug 04 18:20:42.914633 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:42.914464    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:42.915524 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:42.915506    2543 generic.go:356] "Generic (PLEG): container finished" podID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerID="def66af1643bd60fa2f8d60aaae2e49aeffe56300b25395678f8e08768c48bce" exitCode=0
Aug 04 18:20:42.915576 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:42.915555    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" event={"ID":"6ea85fc4-2d8b-43f8-90e2-e380068b629c","Type":"ContainerDied","Data":"def66af1643bd60fa2f8d60aaae2e49aeffe56300b25395678f8e08768c48bce"}
Aug 04 18:20:42.935068 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:42.935030    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" podStartSLOduration=2.473468408 podStartE2EDuration="5.935016779s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.461548371s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:20:42.933080715 +0000 UTC m=+1245.903725067" watchObservedRunningTime="2026-08-04 18:20:42.935016779 +0000 UTC m=+1245.905661118"
Aug 04 18:20:43.921806 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:43.921771    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" event={"ID":"6ea85fc4-2d8b-43f8-90e2-e380068b629c","Type":"ContainerStarted","Data":"ed6c6e71f8b14936751645b271cda957f678af4aeb379f070b0a2d6c9dadee15"}
Aug 04 18:20:43.922239 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:43.921835    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" event={"ID":"6ea85fc4-2d8b-43f8-90e2-e380068b629c","Type":"ContainerStarted","Data":"f09953d858f1d85064c24e3172e9293cc7a7c25cb2a0b6caa3c4cfd63cb29511"}
Aug 04 18:20:43.922239 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:43.922064    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:43.922386 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:43.922340    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:43.922496 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:43.922476    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:43.923121 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:43.923097    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:20:43.923456 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:43.923433    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:20:43.943041 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:43.943003    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" podStartSLOduration=3.286539816 podStartE2EDuration="6.942988657s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.656448841s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:20:43.942344486 +0000 UTC m=+1246.912988827" watchObservedRunningTime="2026-08-04 18:20:43.942988657 +0000 UTC m=+1246.913632997"
Aug 04 18:20:44.926091 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:44.926051    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:20:44.926480 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:44.926203    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:20:49.930119 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:49.930084    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:20:49.930552 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:49.930503    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:20:49.930612 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:49.930587    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:20:49.930959 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:49.930938    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:20:59.930656 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:59.930620    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:20:59.931013 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:20:59.930893    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:21:09.930605 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:09.930570    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:21:09.933185 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:09.930839    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:21:19.930503 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:19.930464    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:21:19.930910 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:19.930878    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:21:29.930519 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:29.930490    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:21:29.930902 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:29.930879    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:21:39.930702 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:39.930664    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:21:39.931201 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:39.930876    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:21:49.931359 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:49.931329    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:21:49.931712 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:49.931406    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:21:57.238852 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:57.238817    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"]
Aug 04 18:21:57.239298 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:57.239054    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kserve-container" containerID="cri-o://f09953d858f1d85064c24e3172e9293cc7a7c25cb2a0b6caa3c4cfd63cb29511" gracePeriod=30
Aug 04 18:21:57.239298 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:57.239085    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kube-rbac-proxy" containerID="cri-o://ed6c6e71f8b14936751645b271cda957f678af4aeb379f070b0a2d6c9dadee15" gracePeriod=30
Aug 04 18:21:57.241516 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:57.241493    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"]
Aug 04 18:21:57.241895 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:57.241707    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kserve-container" containerID="cri-o://c8a248c271618d328cb1be397e1924736d1ea434a69871da466d64b8b5f15a58" gracePeriod=30
Aug 04 18:21:57.241895 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:57.241751    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kube-rbac-proxy" containerID="cri-o://f4651340f3b816ee0e099aaf5beeb060b2107f1264197eb2659b2b8c757a099e" gracePeriod=30
Aug 04 18:21:58.210139 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:58.210105    2543 generic.go:356] "Generic (PLEG): container finished" podID="37d12933-dfdc-410a-a449-b016ed34a8af" containerID="f4651340f3b816ee0e099aaf5beeb060b2107f1264197eb2659b2b8c757a099e" exitCode=2
Aug 04 18:21:58.210327 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:58.210173    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" event={"ID":"37d12933-dfdc-410a-a449-b016ed34a8af","Type":"ContainerDied","Data":"f4651340f3b816ee0e099aaf5beeb060b2107f1264197eb2659b2b8c757a099e"}
Aug 04 18:21:58.211996 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:58.211976    2543 generic.go:356] "Generic (PLEG): container finished" podID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerID="ed6c6e71f8b14936751645b271cda957f678af4aeb379f070b0a2d6c9dadee15" exitCode=2
Aug 04 18:21:58.212088 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:58.212017    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" event={"ID":"6ea85fc4-2d8b-43f8-90e2-e380068b629c","Type":"ContainerDied","Data":"ed6c6e71f8b14936751645b271cda957f678af4aeb379f070b0a2d6c9dadee15"}
Aug 04 18:21:59.926802 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:59.926760    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Aug 04 18:21:59.927163 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:59.926774    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Aug 04 18:21:59.931119 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:59.931094    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:21:59.931192 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:21:59.931094    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:22:01.178051 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.178025    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:22:01.231737 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.231681    2543 generic.go:356] "Generic (PLEG): container finished" podID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerID="f09953d858f1d85064c24e3172e9293cc7a7c25cb2a0b6caa3c4cfd63cb29511" exitCode=0
Aug 04 18:22:01.231737 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.231722    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" event={"ID":"6ea85fc4-2d8b-43f8-90e2-e380068b629c","Type":"ContainerDied","Data":"f09953d858f1d85064c24e3172e9293cc7a7c25cb2a0b6caa3c4cfd63cb29511"}
Aug 04 18:22:01.233630 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.233607    2543 generic.go:356] "Generic (PLEG): container finished" podID="37d12933-dfdc-410a-a449-b016ed34a8af" containerID="c8a248c271618d328cb1be397e1924736d1ea434a69871da466d64b8b5f15a58" exitCode=0
Aug 04 18:22:01.233738 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.233642    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" event={"ID":"37d12933-dfdc-410a-a449-b016ed34a8af","Type":"ContainerDied","Data":"c8a248c271618d328cb1be397e1924736d1ea434a69871da466d64b8b5f15a58"}
Aug 04 18:22:01.233738 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.233666    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"
Aug 04 18:22:01.233738 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.233678    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8" event={"ID":"37d12933-dfdc-410a-a449-b016ed34a8af","Type":"ContainerDied","Data":"416a29ef439d753b1dd8ffdc6a90d00173f98d9d0c180f722cbd6bab52d3c791"}
Aug 04 18:22:01.233738 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.233703    2543 scope.go:117] "RemoveContainer" containerID="f4651340f3b816ee0e099aaf5beeb060b2107f1264197eb2659b2b8c757a099e"
Aug 04 18:22:01.241295 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.241276    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37d12933-dfdc-410a-a449-b016ed34a8af-kube-rbac-proxy-sar-config\") pod \"37d12933-dfdc-410a-a449-b016ed34a8af\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") "
Aug 04 18:22:01.241419 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.241325    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37d12933-dfdc-410a-a449-b016ed34a8af-cabundle-cert\") pod \"37d12933-dfdc-410a-a449-b016ed34a8af\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") "
Aug 04 18:22:01.241419 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.241345    2543 scope.go:117] "RemoveContainer" containerID="c8a248c271618d328cb1be397e1924736d1ea434a69871da466d64b8b5f15a58"
Aug 04 18:22:01.241419 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.241389    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qgmv\" (UniqueName: \"kubernetes.io/projected/37d12933-dfdc-410a-a449-b016ed34a8af-kube-api-access-8qgmv\") pod \"37d12933-dfdc-410a-a449-b016ed34a8af\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") "
Aug 04 18:22:01.241555 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.241446    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37d12933-dfdc-410a-a449-b016ed34a8af-kserve-provision-location\") pod \"37d12933-dfdc-410a-a449-b016ed34a8af\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") "
Aug 04 18:22:01.241555 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.241472    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d12933-dfdc-410a-a449-b016ed34a8af-proxy-tls\") pod \"37d12933-dfdc-410a-a449-b016ed34a8af\" (UID: \"37d12933-dfdc-410a-a449-b016ed34a8af\") "
Aug 04 18:22:01.241740 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.241717    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d12933-dfdc-410a-a449-b016ed34a8af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37d12933-dfdc-410a-a449-b016ed34a8af" (UID: "37d12933-dfdc-410a-a449-b016ed34a8af"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:22:01.241814 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.241794    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d12933-dfdc-410a-a449-b016ed34a8af-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37d12933-dfdc-410a-a449-b016ed34a8af" (UID: "37d12933-dfdc-410a-a449-b016ed34a8af"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:22:01.242040 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.242016    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d12933-dfdc-410a-a449-b016ed34a8af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37d12933-dfdc-410a-a449-b016ed34a8af" (UID: "37d12933-dfdc-410a-a449-b016ed34a8af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:22:01.243610 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.243584    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d12933-dfdc-410a-a449-b016ed34a8af-kube-api-access-8qgmv" (OuterVolumeSpecName: "kube-api-access-8qgmv") pod "37d12933-dfdc-410a-a449-b016ed34a8af" (UID: "37d12933-dfdc-410a-a449-b016ed34a8af"). InnerVolumeSpecName "kube-api-access-8qgmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:22:01.243803 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.243785    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d12933-dfdc-410a-a449-b016ed34a8af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37d12933-dfdc-410a-a449-b016ed34a8af" (UID: "37d12933-dfdc-410a-a449-b016ed34a8af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:22:01.252790 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.252769    2543 scope.go:117] "RemoveContainer" containerID="1deedc210307d67c24bacb403a9b29ababb76153b1bcedd5740a9703e509b17c"
Aug 04 18:22:01.259613 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.259590    2543 scope.go:117] "RemoveContainer" containerID="f4651340f3b816ee0e099aaf5beeb060b2107f1264197eb2659b2b8c757a099e"
Aug 04 18:22:01.259862 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:22:01.259842    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4651340f3b816ee0e099aaf5beeb060b2107f1264197eb2659b2b8c757a099e\": container with ID starting with f4651340f3b816ee0e099aaf5beeb060b2107f1264197eb2659b2b8c757a099e not found: ID does not exist" containerID="f4651340f3b816ee0e099aaf5beeb060b2107f1264197eb2659b2b8c757a099e"
Aug 04 18:22:01.259910 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.259869    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4651340f3b816ee0e099aaf5beeb060b2107f1264197eb2659b2b8c757a099e"} err="failed to get container status \"f4651340f3b816ee0e099aaf5beeb060b2107f1264197eb2659b2b8c757a099e\": rpc error: code = NotFound desc = could not find container \"f4651340f3b816ee0e099aaf5beeb060b2107f1264197eb2659b2b8c757a099e\": container with ID starting with f4651340f3b816ee0e099aaf5beeb060b2107f1264197eb2659b2b8c757a099e not found: ID does not exist"
Aug 04 18:22:01.259910 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.259886    2543 scope.go:117] "RemoveContainer" containerID="c8a248c271618d328cb1be397e1924736d1ea434a69871da466d64b8b5f15a58"
Aug 04 18:22:01.260127 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:22:01.260107    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a248c271618d328cb1be397e1924736d1ea434a69871da466d64b8b5f15a58\": container with ID starting with c8a248c271618d328cb1be397e1924736d1ea434a69871da466d64b8b5f15a58 not found: ID does not exist" containerID="c8a248c271618d328cb1be397e1924736d1ea434a69871da466d64b8b5f15a58"
Aug 04 18:22:01.260190 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.260134    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a248c271618d328cb1be397e1924736d1ea434a69871da466d64b8b5f15a58"} err="failed to get container status \"c8a248c271618d328cb1be397e1924736d1ea434a69871da466d64b8b5f15a58\": rpc error: code = NotFound desc = could not find container \"c8a248c271618d328cb1be397e1924736d1ea434a69871da466d64b8b5f15a58\": container with ID starting with c8a248c271618d328cb1be397e1924736d1ea434a69871da466d64b8b5f15a58 not found: ID does not exist"
Aug 04 18:22:01.260190 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.260156    2543 scope.go:117] "RemoveContainer" containerID="1deedc210307d67c24bacb403a9b29ababb76153b1bcedd5740a9703e509b17c"
Aug 04 18:22:01.260402 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:22:01.260387    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1deedc210307d67c24bacb403a9b29ababb76153b1bcedd5740a9703e509b17c\": container with ID starting with 1deedc210307d67c24bacb403a9b29ababb76153b1bcedd5740a9703e509b17c not found: ID does not exist" containerID="1deedc210307d67c24bacb403a9b29ababb76153b1bcedd5740a9703e509b17c"
Aug 04 18:22:01.260451 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.260406    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1deedc210307d67c24bacb403a9b29ababb76153b1bcedd5740a9703e509b17c"} err="failed to get container status \"1deedc210307d67c24bacb403a9b29ababb76153b1bcedd5740a9703e509b17c\": rpc error: code = NotFound desc = could not find container \"1deedc210307d67c24bacb403a9b29ababb76153b1bcedd5740a9703e509b17c\": container with ID starting with 1deedc210307d67c24bacb403a9b29ababb76153b1bcedd5740a9703e509b17c not found: ID does not exist"
Aug 04 18:22:01.282014 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.281993    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:22:01.342122 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.342098    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kube-rbac-proxy-sar-config\") pod \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") "
Aug 04 18:22:01.342229 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.342130    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kserve-provision-location\") pod \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") "
Aug 04 18:22:01.342229 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.342177    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6q4k\" (UniqueName: \"kubernetes.io/projected/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kube-api-access-q6q4k\") pod \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") "
Aug 04 18:22:01.342368 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.342234    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea85fc4-2d8b-43f8-90e2-e380068b629c-proxy-tls\") pod \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") "
Aug 04 18:22:01.342368 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.342260    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ea85fc4-2d8b-43f8-90e2-e380068b629c-cabundle-cert\") pod \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\" (UID: \"6ea85fc4-2d8b-43f8-90e2-e380068b629c\") "
Aug 04 18:22:01.342501 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.342412    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ea85fc4-2d8b-43f8-90e2-e380068b629c" (UID: "6ea85fc4-2d8b-43f8-90e2-e380068b629c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:22:01.342501 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.342496    2543 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37d12933-dfdc-410a-a449-b016ed34a8af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:22:01.342595 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.342513    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37d12933-dfdc-410a-a449-b016ed34a8af-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:22:01.342595 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.342527    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qgmv\" (UniqueName: \"kubernetes.io/projected/37d12933-dfdc-410a-a449-b016ed34a8af-kube-api-access-8qgmv\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:22:01.342595 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.342541    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37d12933-dfdc-410a-a449-b016ed34a8af-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:22:01.342595 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.342541    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ea85fc4-2d8b-43f8-90e2-e380068b629c" (UID: "6ea85fc4-2d8b-43f8-90e2-e380068b629c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:22:01.342595 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.342554    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d12933-dfdc-410a-a449-b016ed34a8af-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:22:01.342781 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.342620    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea85fc4-2d8b-43f8-90e2-e380068b629c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ea85fc4-2d8b-43f8-90e2-e380068b629c" (UID: "6ea85fc4-2d8b-43f8-90e2-e380068b629c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:22:01.344197 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.344160    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kube-api-access-q6q4k" (OuterVolumeSpecName: "kube-api-access-q6q4k") pod "6ea85fc4-2d8b-43f8-90e2-e380068b629c" (UID: "6ea85fc4-2d8b-43f8-90e2-e380068b629c"). InnerVolumeSpecName "kube-api-access-q6q4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:22:01.344197 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.344174    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea85fc4-2d8b-43f8-90e2-e380068b629c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ea85fc4-2d8b-43f8-90e2-e380068b629c" (UID: "6ea85fc4-2d8b-43f8-90e2-e380068b629c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:22:01.442859 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.442836    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea85fc4-2d8b-43f8-90e2-e380068b629c-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:22:01.442859 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.442855    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ea85fc4-2d8b-43f8-90e2-e380068b629c-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:22:01.442970 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.442865    2543 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:22:01.442970 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.442876    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:22:01.442970 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.442884    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6q4k\" (UniqueName: \"kubernetes.io/projected/6ea85fc4-2d8b-43f8-90e2-e380068b629c-kube-api-access-q6q4k\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:22:01.557070 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.557047    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"]
Aug 04 18:22:01.566389 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.566367    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-2bmm8"]
Aug 04 18:22:01.573812 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:01.573794    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" path="/var/lib/kubelet/pods/37d12933-dfdc-410a-a449-b016ed34a8af/volumes"
Aug 04 18:22:02.238903 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.238857    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"
Aug 04 18:22:02.239244 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.238854    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m" event={"ID":"6ea85fc4-2d8b-43f8-90e2-e380068b629c","Type":"ContainerDied","Data":"39c656cebbc4714b5fa2a66fa9de07a872f4ddeda0959af093c27e0f28846de0"}
Aug 04 18:22:02.239244 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.239041    2543 scope.go:117] "RemoveContainer" containerID="ed6c6e71f8b14936751645b271cda957f678af4aeb379f070b0a2d6c9dadee15"
Aug 04 18:22:02.246588 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.246570    2543 scope.go:117] "RemoveContainer" containerID="f09953d858f1d85064c24e3172e9293cc7a7c25cb2a0b6caa3c4cfd63cb29511"
Aug 04 18:22:02.253231 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.253214    2543 scope.go:117] "RemoveContainer" containerID="def66af1643bd60fa2f8d60aaae2e49aeffe56300b25395678f8e08768c48bce"
Aug 04 18:22:02.271074 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.271047    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"]
Aug 04 18:22:02.274048 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.274029    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-dpg7m"]
Aug 04 18:22:02.472948 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.472914    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"]
Aug 04 18:22:02.473188 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473176    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kserve-container"
Aug 04 18:22:02.473234 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473189    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kserve-container"
Aug 04 18:22:02.473234 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473198    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kube-rbac-proxy"
Aug 04 18:22:02.473234 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473204    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kube-rbac-proxy"
Aug 04 18:22:02.473234 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473212    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kube-rbac-proxy"
Aug 04 18:22:02.473234 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473218    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kube-rbac-proxy"
Aug 04 18:22:02.473234 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473224    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kserve-container"
Aug 04 18:22:02.473234 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473229    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kserve-container"
Aug 04 18:22:02.473234 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473235    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="storage-initializer"
Aug 04 18:22:02.473515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473241    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="storage-initializer"
Aug 04 18:22:02.473515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473247    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kube-rbac-proxy"
Aug 04 18:22:02.473515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473252    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kube-rbac-proxy"
Aug 04 18:22:02.473515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473259    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="storage-initializer"
Aug 04 18:22:02.473515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473265    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="storage-initializer"
Aug 04 18:22:02.473515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473278    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="storage-initializer"
Aug 04 18:22:02.473515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473284    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="storage-initializer"
Aug 04 18:22:02.473515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473292    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container"
Aug 04 18:22:02.473515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473297    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container"
Aug 04 18:22:02.473515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473369    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kube-rbac-proxy"
Aug 04 18:22:02.473515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473378    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" containerName="kserve-container"
Aug 04 18:22:02.473515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473384    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kube-rbac-proxy"
Aug 04 18:22:02.473515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473390    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kserve-container"
Aug 04 18:22:02.473515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473395    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d12933-dfdc-410a-a449-b016ed34a8af" containerName="kserve-container"
Aug 04 18:22:02.473515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.473402    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b32d8a5-1a56-406a-8a30-5bde553afe4f" containerName="kube-rbac-proxy"
Aug 04 18:22:02.478210 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.478193    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.480821 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.480802    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:22:02.480911 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.480855    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Aug 04 18:22:02.481082 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.481070    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:22:02.481228 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.481200    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Aug 04 18:22:02.481466 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.481450    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:22:02.481538 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.481451    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:22:02.481675 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.481662    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vsp4f\""
Aug 04 18:22:02.487533 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.487516    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"]
Aug 04 18:22:02.548946 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.548896    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8406bb1c-cc36-476b-8578-c99a71d8826d-cabundle-cert\") pod \"isvc-canary-promote-predictor-79fd8cbff8-tbrjs\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.548946 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.548925    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8406bb1c-cc36-476b-8578-c99a71d8826d-proxy-tls\") pod \"isvc-canary-promote-predictor-79fd8cbff8-tbrjs\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.549057 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.548945    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8406bb1c-cc36-476b-8578-c99a71d8826d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-79fd8cbff8-tbrjs\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.549057 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.549004    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mldf4\" (UniqueName: \"kubernetes.io/projected/8406bb1c-cc36-476b-8578-c99a71d8826d-kube-api-access-mldf4\") pod \"isvc-canary-promote-predictor-79fd8cbff8-tbrjs\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.549120 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.549081    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8406bb1c-cc36-476b-8578-c99a71d8826d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-79fd8cbff8-tbrjs\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.649950 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.649926    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8406bb1c-cc36-476b-8578-c99a71d8826d-cabundle-cert\") pod \"isvc-canary-promote-predictor-79fd8cbff8-tbrjs\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.649950 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.649951    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8406bb1c-cc36-476b-8578-c99a71d8826d-proxy-tls\") pod \"isvc-canary-promote-predictor-79fd8cbff8-tbrjs\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.650089 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.649971    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8406bb1c-cc36-476b-8578-c99a71d8826d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-79fd8cbff8-tbrjs\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.650089 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.649993    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mldf4\" (UniqueName: \"kubernetes.io/projected/8406bb1c-cc36-476b-8578-c99a71d8826d-kube-api-access-mldf4\") pod \"isvc-canary-promote-predictor-79fd8cbff8-tbrjs\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.650089 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.650022    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8406bb1c-cc36-476b-8578-c99a71d8826d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-79fd8cbff8-tbrjs\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.650422 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.650399    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8406bb1c-cc36-476b-8578-c99a71d8826d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-79fd8cbff8-tbrjs\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.650679 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.650658    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8406bb1c-cc36-476b-8578-c99a71d8826d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-79fd8cbff8-tbrjs\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.650731 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.650658    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8406bb1c-cc36-476b-8578-c99a71d8826d-cabundle-cert\") pod \"isvc-canary-promote-predictor-79fd8cbff8-tbrjs\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.652342 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.652323    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8406bb1c-cc36-476b-8578-c99a71d8826d-proxy-tls\") pod \"isvc-canary-promote-predictor-79fd8cbff8-tbrjs\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.658883 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.658862    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mldf4\" (UniqueName: \"kubernetes.io/projected/8406bb1c-cc36-476b-8578-c99a71d8826d-kube-api-access-mldf4\") pod \"isvc-canary-promote-predictor-79fd8cbff8-tbrjs\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.788738 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.788719    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:02.914283 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:02.914258    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"]
Aug 04 18:22:02.916140 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:22:02.916113    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8406bb1c_cc36_476b_8578_c99a71d8826d.slice/crio-f3c2b9b109584f119571b468a5f94485038d276a6c42f7ad264d9fad4d0a7b8d WatchSource:0}: Error finding container f3c2b9b109584f119571b468a5f94485038d276a6c42f7ad264d9fad4d0a7b8d: Status 404 returned error can't find the container with id f3c2b9b109584f119571b468a5f94485038d276a6c42f7ad264d9fad4d0a7b8d
Aug 04 18:22:03.245554 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:03.245518    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" event={"ID":"8406bb1c-cc36-476b-8578-c99a71d8826d","Type":"ContainerStarted","Data":"53876caed6b82873923aef03625d78145238cd3d5ee12b9597c3f3141385d9fa"}
Aug 04 18:22:03.245922 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:03.245596    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" event={"ID":"8406bb1c-cc36-476b-8578-c99a71d8826d","Type":"ContainerStarted","Data":"f3c2b9b109584f119571b468a5f94485038d276a6c42f7ad264d9fad4d0a7b8d"}
Aug 04 18:22:03.574436 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:03.574366    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ea85fc4-2d8b-43f8-90e2-e380068b629c" path="/var/lib/kubelet/pods/6ea85fc4-2d8b-43f8-90e2-e380068b629c/volumes"
Aug 04 18:22:07.267808 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:07.267777    2543 generic.go:356] "Generic (PLEG): container finished" podID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerID="53876caed6b82873923aef03625d78145238cd3d5ee12b9597c3f3141385d9fa" exitCode=0
Aug 04 18:22:07.268242 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:07.267833    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" event={"ID":"8406bb1c-cc36-476b-8578-c99a71d8826d","Type":"ContainerDied","Data":"53876caed6b82873923aef03625d78145238cd3d5ee12b9597c3f3141385d9fa"}
Aug 04 18:22:08.273691 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:08.273655    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" event={"ID":"8406bb1c-cc36-476b-8578-c99a71d8826d","Type":"ContainerStarted","Data":"52048b7eeae07c2eeb3d258f09b994cb433e71dac5e87f1e477c2d190dc393df"}
Aug 04 18:22:08.274127 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:08.273720    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" event={"ID":"8406bb1c-cc36-476b-8578-c99a71d8826d","Type":"ContainerStarted","Data":"746fa8c55aaadf8a62e2fce7e18610361aa2b50577b000fbfa65b9ac533726d6"}
Aug 04 18:22:08.274127 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:08.273943    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:08.296229 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:08.296191    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" podStartSLOduration=2.62009947 podStartE2EDuration="6.29618008s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.67608061s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:22:08.294432942 +0000 UTC m=+1331.265077303" watchObservedRunningTime="2026-08-04 18:22:08.29618008 +0000 UTC m=+1331.266824419"
Aug 04 18:22:09.277944 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:09.277916    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:09.278980 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:09.278955    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 04 18:22:10.282323 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:10.282285    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 04 18:22:15.286737 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:15.286704    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:22:15.287330 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:15.287287    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 04 18:22:25.287659 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:25.287619    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 04 18:22:35.287704 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:35.287667    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 04 18:22:45.287742 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:45.287703    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 04 18:22:55.287273 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:22:55.287239    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 04 18:23:05.288064 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:05.288024    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 04 18:23:15.288445 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:15.288412    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:23:27.489665 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.489636    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"]
Aug 04 18:23:27.490196 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.489898    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kserve-container" containerID="cri-o://746fa8c55aaadf8a62e2fce7e18610361aa2b50577b000fbfa65b9ac533726d6" gracePeriod=30
Aug 04 18:23:27.490196 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.489955    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kube-rbac-proxy" containerID="cri-o://52048b7eeae07c2eeb3d258f09b994cb433e71dac5e87f1e477c2d190dc393df" gracePeriod=30
Aug 04 18:23:27.532890 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.532860    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"]
Aug 04 18:23:27.535835 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.535817    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.538378 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.538359    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Aug 04 18:23:27.547649 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.547630    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"]
Aug 04 18:23:27.639962 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.639939    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svdp4\" (UniqueName: \"kubernetes.io/projected/11d127af-82b9-458c-82c7-0d71005c4dc2-kube-api-access-svdp4\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-v5d25\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.640029 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.639970    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11d127af-82b9-458c-82c7-0d71005c4dc2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-v5d25\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.640029 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.639995    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11d127af-82b9-458c-82c7-0d71005c4dc2-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-v5d25\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.640113 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.640063    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11d127af-82b9-458c-82c7-0d71005c4dc2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-v5d25\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.640113 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.640100    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11d127af-82b9-458c-82c7-0d71005c4dc2-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-v5d25\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.740436 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.740389    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11d127af-82b9-458c-82c7-0d71005c4dc2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-v5d25\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.740436 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.740426    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11d127af-82b9-458c-82c7-0d71005c4dc2-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-v5d25\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.740550 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.740463    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11d127af-82b9-458c-82c7-0d71005c4dc2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-v5d25\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.740550 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.740502    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11d127af-82b9-458c-82c7-0d71005c4dc2-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-v5d25\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.740636 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.740558    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svdp4\" (UniqueName: \"kubernetes.io/projected/11d127af-82b9-458c-82c7-0d71005c4dc2-kube-api-access-svdp4\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-v5d25\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.740896 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.740870    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11d127af-82b9-458c-82c7-0d71005c4dc2-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-v5d25\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.741080 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.741064    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11d127af-82b9-458c-82c7-0d71005c4dc2-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-v5d25\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.741126 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.741075    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11d127af-82b9-458c-82c7-0d71005c4dc2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-v5d25\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.742518 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.742502    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11d127af-82b9-458c-82c7-0d71005c4dc2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-v5d25\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.749765 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.749747    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svdp4\" (UniqueName: \"kubernetes.io/projected/11d127af-82b9-458c-82c7-0d71005c4dc2-kube-api-access-svdp4\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-v5d25\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.844593 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.844577    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:27.971410 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:27.971387    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"]
Aug 04 18:23:27.973464 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:23:27.973434    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d127af_82b9_458c_82c7_0d71005c4dc2.slice/crio-e0e4323a17f65b6d7245630c27f5e37a3e2d569f5ee85fafaf77a6b6cc23f82a WatchSource:0}: Error finding container e0e4323a17f65b6d7245630c27f5e37a3e2d569f5ee85fafaf77a6b6cc23f82a: Status 404 returned error can't find the container with id e0e4323a17f65b6d7245630c27f5e37a3e2d569f5ee85fafaf77a6b6cc23f82a
Aug 04 18:23:28.579888 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:28.579853    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" event={"ID":"11d127af-82b9-458c-82c7-0d71005c4dc2","Type":"ContainerStarted","Data":"3f6243bfcd505f58c8fb12f81e936ddbc0a5358dc2dca7a9dad011ed4718f5d6"}
Aug 04 18:23:28.580306 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:28.579933    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" event={"ID":"11d127af-82b9-458c-82c7-0d71005c4dc2","Type":"ContainerStarted","Data":"e0e4323a17f65b6d7245630c27f5e37a3e2d569f5ee85fafaf77a6b6cc23f82a"}
Aug 04 18:23:28.581924 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:28.581901    2543 generic.go:356] "Generic (PLEG): container finished" podID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerID="52048b7eeae07c2eeb3d258f09b994cb433e71dac5e87f1e477c2d190dc393df" exitCode=2
Aug 04 18:23:28.582016 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:28.581947    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" event={"ID":"8406bb1c-cc36-476b-8578-c99a71d8826d","Type":"ContainerDied","Data":"52048b7eeae07c2eeb3d258f09b994cb433e71dac5e87f1e477c2d190dc393df"}
Aug 04 18:23:30.282919 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:30.282879    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Aug 04 18:23:31.224443 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.224422    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:23:31.264977 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.264940    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8406bb1c-cc36-476b-8578-c99a71d8826d-kserve-provision-location\") pod \"8406bb1c-cc36-476b-8578-c99a71d8826d\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") "
Aug 04 18:23:31.265093 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.265010    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8406bb1c-cc36-476b-8578-c99a71d8826d-proxy-tls\") pod \"8406bb1c-cc36-476b-8578-c99a71d8826d\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") "
Aug 04 18:23:31.265093 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.265064    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8406bb1c-cc36-476b-8578-c99a71d8826d-cabundle-cert\") pod \"8406bb1c-cc36-476b-8578-c99a71d8826d\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") "
Aug 04 18:23:31.265188 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.265104    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mldf4\" (UniqueName: \"kubernetes.io/projected/8406bb1c-cc36-476b-8578-c99a71d8826d-kube-api-access-mldf4\") pod \"8406bb1c-cc36-476b-8578-c99a71d8826d\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") "
Aug 04 18:23:31.265188 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.265166    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8406bb1c-cc36-476b-8578-c99a71d8826d-kube-rbac-proxy-sar-config\") pod \"8406bb1c-cc36-476b-8578-c99a71d8826d\" (UID: \"8406bb1c-cc36-476b-8578-c99a71d8826d\") "
Aug 04 18:23:31.265263 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.265208    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8406bb1c-cc36-476b-8578-c99a71d8826d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8406bb1c-cc36-476b-8578-c99a71d8826d" (UID: "8406bb1c-cc36-476b-8578-c99a71d8826d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:23:31.265443 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.265424    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8406bb1c-cc36-476b-8578-c99a71d8826d-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:23:31.265549 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.265526    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8406bb1c-cc36-476b-8578-c99a71d8826d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8406bb1c-cc36-476b-8578-c99a71d8826d" (UID: "8406bb1c-cc36-476b-8578-c99a71d8826d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:23:31.265608 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.265587    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8406bb1c-cc36-476b-8578-c99a71d8826d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8406bb1c-cc36-476b-8578-c99a71d8826d" (UID: "8406bb1c-cc36-476b-8578-c99a71d8826d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:23:31.267007 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.266983    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8406bb1c-cc36-476b-8578-c99a71d8826d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8406bb1c-cc36-476b-8578-c99a71d8826d" (UID: "8406bb1c-cc36-476b-8578-c99a71d8826d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:23:31.267097 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.267045    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8406bb1c-cc36-476b-8578-c99a71d8826d-kube-api-access-mldf4" (OuterVolumeSpecName: "kube-api-access-mldf4") pod "8406bb1c-cc36-476b-8578-c99a71d8826d" (UID: "8406bb1c-cc36-476b-8578-c99a71d8826d"). InnerVolumeSpecName "kube-api-access-mldf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:23:31.366491 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.366467    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mldf4\" (UniqueName: \"kubernetes.io/projected/8406bb1c-cc36-476b-8578-c99a71d8826d-kube-api-access-mldf4\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:23:31.366491 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.366490    2543 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8406bb1c-cc36-476b-8578-c99a71d8826d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:23:31.366788 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.366504    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8406bb1c-cc36-476b-8578-c99a71d8826d-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:23:31.366788 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.366517    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8406bb1c-cc36-476b-8578-c99a71d8826d-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:23:31.593942 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.593923    2543 generic.go:356] "Generic (PLEG): container finished" podID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerID="746fa8c55aaadf8a62e2fce7e18610361aa2b50577b000fbfa65b9ac533726d6" exitCode=0
Aug 04 18:23:31.594019 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.593976    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"
Aug 04 18:23:31.594019 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.593981    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" event={"ID":"8406bb1c-cc36-476b-8578-c99a71d8826d","Type":"ContainerDied","Data":"746fa8c55aaadf8a62e2fce7e18610361aa2b50577b000fbfa65b9ac533726d6"}
Aug 04 18:23:31.594093 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.594021    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs" event={"ID":"8406bb1c-cc36-476b-8578-c99a71d8826d","Type":"ContainerDied","Data":"f3c2b9b109584f119571b468a5f94485038d276a6c42f7ad264d9fad4d0a7b8d"}
Aug 04 18:23:31.594093 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.594045    2543 scope.go:117] "RemoveContainer" containerID="52048b7eeae07c2eeb3d258f09b994cb433e71dac5e87f1e477c2d190dc393df"
Aug 04 18:23:31.603771 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.603756    2543 scope.go:117] "RemoveContainer" containerID="746fa8c55aaadf8a62e2fce7e18610361aa2b50577b000fbfa65b9ac533726d6"
Aug 04 18:23:31.610477 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.610448    2543 scope.go:117] "RemoveContainer" containerID="53876caed6b82873923aef03625d78145238cd3d5ee12b9597c3f3141385d9fa"
Aug 04 18:23:31.612893 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.612875    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"]
Aug 04 18:23:31.617409 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.617368    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-tbrjs"]
Aug 04 18:23:31.617811 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.617800    2543 scope.go:117] "RemoveContainer" containerID="52048b7eeae07c2eeb3d258f09b994cb433e71dac5e87f1e477c2d190dc393df"
Aug 04 18:23:31.618040 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:23:31.618022    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52048b7eeae07c2eeb3d258f09b994cb433e71dac5e87f1e477c2d190dc393df\": container with ID starting with 52048b7eeae07c2eeb3d258f09b994cb433e71dac5e87f1e477c2d190dc393df not found: ID does not exist" containerID="52048b7eeae07c2eeb3d258f09b994cb433e71dac5e87f1e477c2d190dc393df"
Aug 04 18:23:31.618119 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.618051    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52048b7eeae07c2eeb3d258f09b994cb433e71dac5e87f1e477c2d190dc393df"} err="failed to get container status \"52048b7eeae07c2eeb3d258f09b994cb433e71dac5e87f1e477c2d190dc393df\": rpc error: code = NotFound desc = could not find container \"52048b7eeae07c2eeb3d258f09b994cb433e71dac5e87f1e477c2d190dc393df\": container with ID starting with 52048b7eeae07c2eeb3d258f09b994cb433e71dac5e87f1e477c2d190dc393df not found: ID does not exist"
Aug 04 18:23:31.618119 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.618073    2543 scope.go:117] "RemoveContainer" containerID="746fa8c55aaadf8a62e2fce7e18610361aa2b50577b000fbfa65b9ac533726d6"
Aug 04 18:23:31.618301 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:23:31.618286    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746fa8c55aaadf8a62e2fce7e18610361aa2b50577b000fbfa65b9ac533726d6\": container with ID starting with 746fa8c55aaadf8a62e2fce7e18610361aa2b50577b000fbfa65b9ac533726d6 not found: ID does not exist" containerID="746fa8c55aaadf8a62e2fce7e18610361aa2b50577b000fbfa65b9ac533726d6"
Aug 04 18:23:31.618363 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.618320    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746fa8c55aaadf8a62e2fce7e18610361aa2b50577b000fbfa65b9ac533726d6"} err="failed to get container status \"746fa8c55aaadf8a62e2fce7e18610361aa2b50577b000fbfa65b9ac533726d6\": rpc error: code = NotFound desc = could not find container \"746fa8c55aaadf8a62e2fce7e18610361aa2b50577b000fbfa65b9ac533726d6\": container with ID starting with 746fa8c55aaadf8a62e2fce7e18610361aa2b50577b000fbfa65b9ac533726d6 not found: ID does not exist"
Aug 04 18:23:31.618363 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.618338    2543 scope.go:117] "RemoveContainer" containerID="53876caed6b82873923aef03625d78145238cd3d5ee12b9597c3f3141385d9fa"
Aug 04 18:23:31.618551 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:23:31.618536    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53876caed6b82873923aef03625d78145238cd3d5ee12b9597c3f3141385d9fa\": container with ID starting with 53876caed6b82873923aef03625d78145238cd3d5ee12b9597c3f3141385d9fa not found: ID does not exist" containerID="53876caed6b82873923aef03625d78145238cd3d5ee12b9597c3f3141385d9fa"
Aug 04 18:23:31.618594 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:31.618554    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53876caed6b82873923aef03625d78145238cd3d5ee12b9597c3f3141385d9fa"} err="failed to get container status \"53876caed6b82873923aef03625d78145238cd3d5ee12b9597c3f3141385d9fa\": rpc error: code = NotFound desc = could not find container \"53876caed6b82873923aef03625d78145238cd3d5ee12b9597c3f3141385d9fa\": container with ID starting with 53876caed6b82873923aef03625d78145238cd3d5ee12b9597c3f3141385d9fa not found: ID does not exist"
Aug 04 18:23:32.600122 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:32.600093    2543 generic.go:356] "Generic (PLEG): container finished" podID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerID="3f6243bfcd505f58c8fb12f81e936ddbc0a5358dc2dca7a9dad011ed4718f5d6" exitCode=0
Aug 04 18:23:32.600535 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:32.600147    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" event={"ID":"11d127af-82b9-458c-82c7-0d71005c4dc2","Type":"ContainerDied","Data":"3f6243bfcd505f58c8fb12f81e936ddbc0a5358dc2dca7a9dad011ed4718f5d6"}
Aug 04 18:23:33.574301 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:33.574267    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" path="/var/lib/kubelet/pods/8406bb1c-cc36-476b-8578-c99a71d8826d/volumes"
Aug 04 18:23:33.606616 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:33.606593    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" event={"ID":"11d127af-82b9-458c-82c7-0d71005c4dc2","Type":"ContainerStarted","Data":"a2305ca66113273448c3ca43e2c93d897b7ef5cdef85d694fe01e9d16dac0aac"}
Aug 04 18:23:33.606963 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:33.606668    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" event={"ID":"11d127af-82b9-458c-82c7-0d71005c4dc2","Type":"ContainerStarted","Data":"9261f9d10e563205cf49537fea5469445cf0544cdd120e63610e2bb831188350"}
Aug 04 18:23:33.606963 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:33.606866    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:33.627364 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:33.627299    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" podStartSLOduration=2.759546815 podStartE2EDuration="6.627286009s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.867739194s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:23:33.625501769 +0000 UTC m=+1416.596146121" watchObservedRunningTime="2026-08-04 18:23:33.627286009 +0000 UTC m=+1416.597930394"
Aug 04 18:23:34.610583 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:34.610555    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:34.611538 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:34.611512    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:23:35.614458 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:35.614421    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:23:40.618376 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:40.618346    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:23:40.618838 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:40.618814    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:23:50.619676 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:23:50.619599    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:24:00.618746 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:00.618707    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:24:10.618748 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:10.618714    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:24:20.618901 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:20.618866    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:24:30.619520 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:30.619483    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:24:40.619446 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:40.619419    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:24:47.794573 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:47.794539    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"]
Aug 04 18:24:47.795024 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:47.794810    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kserve-container" containerID="cri-o://9261f9d10e563205cf49537fea5469445cf0544cdd120e63610e2bb831188350" gracePeriod=30
Aug 04 18:24:47.795024 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:47.794858    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kube-rbac-proxy" containerID="cri-o://a2305ca66113273448c3ca43e2c93d897b7ef5cdef85d694fe01e9d16dac0aac" gracePeriod=30
Aug 04 18:24:48.887535 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:48.887500    2543 generic.go:356] "Generic (PLEG): container finished" podID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerID="a2305ca66113273448c3ca43e2c93d897b7ef5cdef85d694fe01e9d16dac0aac" exitCode=2
Aug 04 18:24:48.887888 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:48.887545    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" event={"ID":"11d127af-82b9-458c-82c7-0d71005c4dc2","Type":"ContainerDied","Data":"a2305ca66113273448c3ca43e2c93d897b7ef5cdef85d694fe01e9d16dac0aac"}
Aug 04 18:24:50.615562 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:50.615526    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Aug 04 18:24:50.618816 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:50.618787    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:24:51.234485 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.234463    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:24:51.315744 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.315679    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11d127af-82b9-458c-82c7-0d71005c4dc2-proxy-tls\") pod \"11d127af-82b9-458c-82c7-0d71005c4dc2\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") "
Aug 04 18:24:51.315861 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.315758    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11d127af-82b9-458c-82c7-0d71005c4dc2-kserve-provision-location\") pod \"11d127af-82b9-458c-82c7-0d71005c4dc2\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") "
Aug 04 18:24:51.315861 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.315791    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11d127af-82b9-458c-82c7-0d71005c4dc2-kube-rbac-proxy-sar-config\") pod \"11d127af-82b9-458c-82c7-0d71005c4dc2\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") "
Aug 04 18:24:51.315861 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.315819    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svdp4\" (UniqueName: \"kubernetes.io/projected/11d127af-82b9-458c-82c7-0d71005c4dc2-kube-api-access-svdp4\") pod \"11d127af-82b9-458c-82c7-0d71005c4dc2\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") "
Aug 04 18:24:51.315861 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.315838    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11d127af-82b9-458c-82c7-0d71005c4dc2-cabundle-cert\") pod \"11d127af-82b9-458c-82c7-0d71005c4dc2\" (UID: \"11d127af-82b9-458c-82c7-0d71005c4dc2\") "
Aug 04 18:24:51.316080 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.316044    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d127af-82b9-458c-82c7-0d71005c4dc2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "11d127af-82b9-458c-82c7-0d71005c4dc2" (UID: "11d127af-82b9-458c-82c7-0d71005c4dc2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:24:51.316177 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.316150    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d127af-82b9-458c-82c7-0d71005c4dc2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "11d127af-82b9-458c-82c7-0d71005c4dc2" (UID: "11d127af-82b9-458c-82c7-0d71005c4dc2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:24:51.316229 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.316212    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d127af-82b9-458c-82c7-0d71005c4dc2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "11d127af-82b9-458c-82c7-0d71005c4dc2" (UID: "11d127af-82b9-458c-82c7-0d71005c4dc2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:24:51.317718 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.317688    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d127af-82b9-458c-82c7-0d71005c4dc2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "11d127af-82b9-458c-82c7-0d71005c4dc2" (UID: "11d127af-82b9-458c-82c7-0d71005c4dc2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:24:51.317718 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.317700    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d127af-82b9-458c-82c7-0d71005c4dc2-kube-api-access-svdp4" (OuterVolumeSpecName: "kube-api-access-svdp4") pod "11d127af-82b9-458c-82c7-0d71005c4dc2" (UID: "11d127af-82b9-458c-82c7-0d71005c4dc2"). InnerVolumeSpecName "kube-api-access-svdp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:24:51.416798 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.416770    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11d127af-82b9-458c-82c7-0d71005c4dc2-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:24:51.416798 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.416795    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11d127af-82b9-458c-82c7-0d71005c4dc2-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:24:51.416798 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.416804    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11d127af-82b9-458c-82c7-0d71005c4dc2-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:24:51.416955 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.416814    2543 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11d127af-82b9-458c-82c7-0d71005c4dc2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:24:51.416955 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.416822    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svdp4\" (UniqueName: \"kubernetes.io/projected/11d127af-82b9-458c-82c7-0d71005c4dc2-kube-api-access-svdp4\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:24:51.899515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.899492    2543 generic.go:356] "Generic (PLEG): container finished" podID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerID="9261f9d10e563205cf49537fea5469445cf0544cdd120e63610e2bb831188350" exitCode=0
Aug 04 18:24:51.899804 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.899541    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"
Aug 04 18:24:51.899804 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.899562    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" event={"ID":"11d127af-82b9-458c-82c7-0d71005c4dc2","Type":"ContainerDied","Data":"9261f9d10e563205cf49537fea5469445cf0544cdd120e63610e2bb831188350"}
Aug 04 18:24:51.899804 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.899604    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25" event={"ID":"11d127af-82b9-458c-82c7-0d71005c4dc2","Type":"ContainerDied","Data":"e0e4323a17f65b6d7245630c27f5e37a3e2d569f5ee85fafaf77a6b6cc23f82a"}
Aug 04 18:24:51.899804 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.899628    2543 scope.go:117] "RemoveContainer" containerID="a2305ca66113273448c3ca43e2c93d897b7ef5cdef85d694fe01e9d16dac0aac"
Aug 04 18:24:51.907036 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.907019    2543 scope.go:117] "RemoveContainer" containerID="9261f9d10e563205cf49537fea5469445cf0544cdd120e63610e2bb831188350"
Aug 04 18:24:51.913963 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.913948    2543 scope.go:117] "RemoveContainer" containerID="3f6243bfcd505f58c8fb12f81e936ddbc0a5358dc2dca7a9dad011ed4718f5d6"
Aug 04 18:24:51.920832 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.920813    2543 scope.go:117] "RemoveContainer" containerID="a2305ca66113273448c3ca43e2c93d897b7ef5cdef85d694fe01e9d16dac0aac"
Aug 04 18:24:51.921080 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:24:51.921062    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2305ca66113273448c3ca43e2c93d897b7ef5cdef85d694fe01e9d16dac0aac\": container with ID starting with a2305ca66113273448c3ca43e2c93d897b7ef5cdef85d694fe01e9d16dac0aac not found: ID does not exist" containerID="a2305ca66113273448c3ca43e2c93d897b7ef5cdef85d694fe01e9d16dac0aac"
Aug 04 18:24:51.921143 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.921086    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2305ca66113273448c3ca43e2c93d897b7ef5cdef85d694fe01e9d16dac0aac"} err="failed to get container status \"a2305ca66113273448c3ca43e2c93d897b7ef5cdef85d694fe01e9d16dac0aac\": rpc error: code = NotFound desc = could not find container \"a2305ca66113273448c3ca43e2c93d897b7ef5cdef85d694fe01e9d16dac0aac\": container with ID starting with a2305ca66113273448c3ca43e2c93d897b7ef5cdef85d694fe01e9d16dac0aac not found: ID does not exist"
Aug 04 18:24:51.921143 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.921103    2543 scope.go:117] "RemoveContainer" containerID="9261f9d10e563205cf49537fea5469445cf0544cdd120e63610e2bb831188350"
Aug 04 18:24:51.921345 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:24:51.921325    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9261f9d10e563205cf49537fea5469445cf0544cdd120e63610e2bb831188350\": container with ID starting with 9261f9d10e563205cf49537fea5469445cf0544cdd120e63610e2bb831188350 not found: ID does not exist" containerID="9261f9d10e563205cf49537fea5469445cf0544cdd120e63610e2bb831188350"
Aug 04 18:24:51.921448 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.921348    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9261f9d10e563205cf49537fea5469445cf0544cdd120e63610e2bb831188350"} err="failed to get container status \"9261f9d10e563205cf49537fea5469445cf0544cdd120e63610e2bb831188350\": rpc error: code = NotFound desc = could not find container \"9261f9d10e563205cf49537fea5469445cf0544cdd120e63610e2bb831188350\": container with ID starting with 9261f9d10e563205cf49537fea5469445cf0544cdd120e63610e2bb831188350 not found: ID does not exist"
Aug 04 18:24:51.921448 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.921363    2543 scope.go:117] "RemoveContainer" containerID="3f6243bfcd505f58c8fb12f81e936ddbc0a5358dc2dca7a9dad011ed4718f5d6"
Aug 04 18:24:51.921609 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:24:51.921594    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6243bfcd505f58c8fb12f81e936ddbc0a5358dc2dca7a9dad011ed4718f5d6\": container with ID starting with 3f6243bfcd505f58c8fb12f81e936ddbc0a5358dc2dca7a9dad011ed4718f5d6 not found: ID does not exist" containerID="3f6243bfcd505f58c8fb12f81e936ddbc0a5358dc2dca7a9dad011ed4718f5d6"
Aug 04 18:24:51.921650 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.921614    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6243bfcd505f58c8fb12f81e936ddbc0a5358dc2dca7a9dad011ed4718f5d6"} err="failed to get container status \"3f6243bfcd505f58c8fb12f81e936ddbc0a5358dc2dca7a9dad011ed4718f5d6\": rpc error: code = NotFound desc = could not find container \"3f6243bfcd505f58c8fb12f81e936ddbc0a5358dc2dca7a9dad011ed4718f5d6\": container with ID starting with 3f6243bfcd505f58c8fb12f81e936ddbc0a5358dc2dca7a9dad011ed4718f5d6 not found: ID does not exist"
Aug 04 18:24:51.925765 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.925746    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"]
Aug 04 18:24:51.931983 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:51.931965    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-v5d25"]
Aug 04 18:24:53.574392 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:53.574355    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" path="/var/lib/kubelet/pods/11d127af-82b9-458c-82c7-0d71005c4dc2/volumes"
Aug 04 18:24:57.552439 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:57.552412    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/1.log"
Aug 04 18:24:57.554401 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:24:57.554373    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/1.log"
Aug 04 18:25:11.585084 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585048    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"]
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585334    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kserve-container"
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585350    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kserve-container"
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585363    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="storage-initializer"
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585369    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="storage-initializer"
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585378    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kube-rbac-proxy"
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585384    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kube-rbac-proxy"
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585394    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kserve-container"
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585399    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kserve-container"
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585412    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kube-rbac-proxy"
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585421    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kube-rbac-proxy"
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585431    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="storage-initializer"
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585437    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="storage-initializer"
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585482    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kube-rbac-proxy"
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585489    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kserve-container"
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585495    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d127af-82b9-458c-82c7-0d71005c4dc2" containerName="kube-rbac-proxy"
Aug 04 18:25:11.585726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.585501    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8406bb1c-cc36-476b-8578-c99a71d8826d" containerName="kserve-container"
Aug 04 18:25:11.588692 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.588665    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:11.591246 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.591202    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:25:11.591408 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.591228    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vsp4f\""
Aug 04 18:25:11.591408 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.591375    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:25:11.591527 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.591418    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Aug 04 18:25:11.591527 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.591479    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Aug 04 18:25:11.591527 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.591523    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:25:11.591677 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.591660    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:25:11.598876 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.598853    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"]
Aug 04 18:25:11.758054 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.758029    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/412e335e-6bcd-4251-af23-971bac722e3d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:11.758175 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.758056    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/412e335e-6bcd-4251-af23-971bac722e3d-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:11.758175 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.758078    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cqd9\" (UniqueName: \"kubernetes.io/projected/412e335e-6bcd-4251-af23-971bac722e3d-kube-api-access-2cqd9\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:11.758175 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.758095    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/412e335e-6bcd-4251-af23-971bac722e3d-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:11.758175 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.758145    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412e335e-6bcd-4251-af23-971bac722e3d-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:11.859371 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.859343    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/412e335e-6bcd-4251-af23-971bac722e3d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:11.859481 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.859381    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/412e335e-6bcd-4251-af23-971bac722e3d-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:11.859481 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.859416    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cqd9\" (UniqueName: \"kubernetes.io/projected/412e335e-6bcd-4251-af23-971bac722e3d-kube-api-access-2cqd9\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:11.859481 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.859447    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/412e335e-6bcd-4251-af23-971bac722e3d-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:11.859642 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.859481    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412e335e-6bcd-4251-af23-971bac722e3d-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:11.859697 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:25:11.859647    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-serving-cert: secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Aug 04 18:25:11.859749 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:25:11.859729    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412e335e-6bcd-4251-af23-971bac722e3d-proxy-tls podName:412e335e-6bcd-4251-af23-971bac722e3d nodeName:}" failed. No retries permitted until 2026-08-04 18:25:12.359708372 +0000 UTC m=+1515.330352692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/412e335e-6bcd-4251-af23-971bac722e3d-proxy-tls") pod "isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" (UID: "412e335e-6bcd-4251-af23-971bac722e3d") : secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Aug 04 18:25:11.859749 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.859738    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/412e335e-6bcd-4251-af23-971bac722e3d-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:11.860095 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.860074    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/412e335e-6bcd-4251-af23-971bac722e3d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:11.860246 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.860230    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/412e335e-6bcd-4251-af23-971bac722e3d-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:11.872014 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:11.871989    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cqd9\" (UniqueName: \"kubernetes.io/projected/412e335e-6bcd-4251-af23-971bac722e3d-kube-api-access-2cqd9\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:12.363019 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:12.363000    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412e335e-6bcd-4251-af23-971bac722e3d-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:12.365075 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:12.365058    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412e335e-6bcd-4251-af23-971bac722e3d-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:12.499254 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:12.499232    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:12.630812 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:12.630790    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"]
Aug 04 18:25:12.633359 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:25:12.633328    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412e335e_6bcd_4251_af23_971bac722e3d.slice/crio-261430379eebe05309fb6fd1bc73be9f9384fc808b1e881990b5f271e3bde400 WatchSource:0}: Error finding container 261430379eebe05309fb6fd1bc73be9f9384fc808b1e881990b5f271e3bde400: Status 404 returned error can't find the container with id 261430379eebe05309fb6fd1bc73be9f9384fc808b1e881990b5f271e3bde400
Aug 04 18:25:12.973385 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:12.973294    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" event={"ID":"412e335e-6bcd-4251-af23-971bac722e3d","Type":"ContainerStarted","Data":"0c79d3913c2154422293feafa7f267fd9bb24725112ea0e560b102c4bcdd266a"}
Aug 04 18:25:12.973385 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:12.973381    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" event={"ID":"412e335e-6bcd-4251-af23-971bac722e3d","Type":"ContainerStarted","Data":"261430379eebe05309fb6fd1bc73be9f9384fc808b1e881990b5f271e3bde400"}
Aug 04 18:25:16.989999 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:16.989970    2543 generic.go:356] "Generic (PLEG): container finished" podID="412e335e-6bcd-4251-af23-971bac722e3d" containerID="0c79d3913c2154422293feafa7f267fd9bb24725112ea0e560b102c4bcdd266a" exitCode=0
Aug 04 18:25:16.990303 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:16.990040    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" event={"ID":"412e335e-6bcd-4251-af23-971bac722e3d","Type":"ContainerDied","Data":"0c79d3913c2154422293feafa7f267fd9bb24725112ea0e560b102c4bcdd266a"}
Aug 04 18:25:17.995508 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:17.995468    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" event={"ID":"412e335e-6bcd-4251-af23-971bac722e3d","Type":"ContainerStarted","Data":"6fb3e8d44e71a2d52573a9ea368373ee91f1800aab5e78b8bc8781347d786ec0"}
Aug 04 18:25:17.995927 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:17.995584    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" event={"ID":"412e335e-6bcd-4251-af23-971bac722e3d","Type":"ContainerStarted","Data":"fedc5d00a246d523c5a9e204259f51815826a9ec060bd6339f6d4b7848c58e8b"}
Aug 04 18:25:17.995927 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:17.995634    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" event={"ID":"412e335e-6bcd-4251-af23-971bac722e3d","Type":"ContainerStarted","Data":"120933e55ab69bd826f059f4df275edef7c57380c0e71d89c6ee01d9135f83a6"}
Aug 04 18:25:17.996410 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:17.996212    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:17.996410 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:17.996278    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:17.996410 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:17.996346    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:17.997675 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:17.997643    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:25:17.998324 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:17.998276    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:25:18.017519 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:18.017479    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podStartSLOduration=3.274313878 podStartE2EDuration="7.017465227s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.743151349s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:25:18.016201946 +0000 UTC m=+1520.986846278" watchObservedRunningTime="2026-08-04 18:25:18.017465227 +0000 UTC m=+1520.988109567"
Aug 04 18:25:18.999929 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:18.999889    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:25:19.000303 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:19.000268    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:25:24.004024 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:24.003997    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:25:24.004483 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:24.004453    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:25:24.004749 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:24.004722    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:25:34.004999 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:34.004956    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:25:34.005514 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:34.005491    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:25:44.005195 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:44.005160    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:25:44.005732 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:44.005666    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:25:54.004897 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:54.004861    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:25:54.005363 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:25:54.005279    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:26:04.005117 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:04.005075    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:26:04.005626 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:04.005606    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:26:14.004500 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:14.004457    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:26:14.006740 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:14.004917    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:26:24.005510 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:24.005474    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:26:24.006095 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:24.005996    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:26:36.895667 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:36.895634    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"]
Aug 04 18:26:36.896079 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:36.895947    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kserve-container" containerID="cri-o://120933e55ab69bd826f059f4df275edef7c57380c0e71d89c6ee01d9135f83a6" gracePeriod=30
Aug 04 18:26:36.896079 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:36.895946    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="agent" containerID="cri-o://6fb3e8d44e71a2d52573a9ea368373ee91f1800aab5e78b8bc8781347d786ec0" gracePeriod=30
Aug 04 18:26:36.896079 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:36.895969    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kube-rbac-proxy" containerID="cri-o://fedc5d00a246d523c5a9e204259f51815826a9ec060bd6339f6d4b7848c58e8b" gracePeriod=30
Aug 04 18:26:37.301263 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:37.301192    2543 generic.go:356] "Generic (PLEG): container finished" podID="412e335e-6bcd-4251-af23-971bac722e3d" containerID="fedc5d00a246d523c5a9e204259f51815826a9ec060bd6339f6d4b7848c58e8b" exitCode=2
Aug 04 18:26:37.301400 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:37.301259    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" event={"ID":"412e335e-6bcd-4251-af23-971bac722e3d","Type":"ContainerDied","Data":"fedc5d00a246d523c5a9e204259f51815826a9ec060bd6339f6d4b7848c58e8b"}
Aug 04 18:26:39.000640 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:39.000595    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Aug 04 18:26:41.318834 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:41.318807    2543 generic.go:356] "Generic (PLEG): container finished" podID="412e335e-6bcd-4251-af23-971bac722e3d" containerID="120933e55ab69bd826f059f4df275edef7c57380c0e71d89c6ee01d9135f83a6" exitCode=0
Aug 04 18:26:41.319159 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:41.318852    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" event={"ID":"412e335e-6bcd-4251-af23-971bac722e3d","Type":"ContainerDied","Data":"120933e55ab69bd826f059f4df275edef7c57380c0e71d89c6ee01d9135f83a6"}
Aug 04 18:26:43.999910 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:43.999873    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Aug 04 18:26:44.005317 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:44.005282    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:26:44.006828 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:44.006800    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:26:49.000471 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:49.000433    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Aug 04 18:26:49.000844 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:49.000565    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:26:52.969773 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:52.969700    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kserve-container/0.log"
Aug 04 18:26:52.994679 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:52.994653    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kube-rbac-proxy/0.log"
Aug 04 18:26:53.010251 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:53.010228    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/agent/0.log"
Aug 04 18:26:53.026726 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:53.026702    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/storage-initializer/0.log"
Aug 04 18:26:53.809146 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:53.809112    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kserve-container/0.log"
Aug 04 18:26:53.823243 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:53.823223    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kube-rbac-proxy/0.log"
Aug 04 18:26:53.838521 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:53.838500    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/agent/0.log"
Aug 04 18:26:53.852337 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:53.852305    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/storage-initializer/0.log"
Aug 04 18:26:54.000022 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:53.999992    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Aug 04 18:26:54.005274 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:54.005254    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:26:54.006873 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:54.006848    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:26:54.655559 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:54.655531    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kserve-container/0.log"
Aug 04 18:26:54.669754 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:54.669733    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kube-rbac-proxy/0.log"
Aug 04 18:26:54.681495 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:54.681475    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/agent/0.log"
Aug 04 18:26:54.693327 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:54.693291    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/storage-initializer/0.log"
Aug 04 18:26:55.465987 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:55.465953    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kserve-container/0.log"
Aug 04 18:26:55.479458 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:55.479435    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kube-rbac-proxy/0.log"
Aug 04 18:26:55.491069 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:55.491051    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/agent/0.log"
Aug 04 18:26:55.502849 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:55.502831    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/storage-initializer/0.log"
Aug 04 18:26:56.283550 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:56.283528    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kserve-container/0.log"
Aug 04 18:26:56.296420 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:56.296403    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kube-rbac-proxy/0.log"
Aug 04 18:26:56.308287 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:56.308267    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/agent/0.log"
Aug 04 18:26:56.320446 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:56.320432    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/storage-initializer/0.log"
Aug 04 18:26:57.139232 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:57.139210    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kserve-container/0.log"
Aug 04 18:26:57.153399 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:57.153382    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kube-rbac-proxy/0.log"
Aug 04 18:26:57.168915 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:57.168898    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/agent/0.log"
Aug 04 18:26:57.188374 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:57.188355    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/storage-initializer/0.log"
Aug 04 18:26:58.033646 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:58.033625    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kserve-container/0.log"
Aug 04 18:26:58.048591 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:58.048562    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kube-rbac-proxy/0.log"
Aug 04 18:26:58.064283 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:58.064266    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/agent/0.log"
Aug 04 18:26:58.078663 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:58.078646    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/storage-initializer/0.log"
Aug 04 18:26:58.904602 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:58.904579    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kserve-container/0.log"
Aug 04 18:26:58.917684 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:58.917658    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kube-rbac-proxy/0.log"
Aug 04 18:26:58.931686 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:58.931665    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/agent/0.log"
Aug 04 18:26:58.943620 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:58.943604    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/storage-initializer/0.log"
Aug 04 18:26:58.999938 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:58.999909    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Aug 04 18:26:59.737072 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:59.737049    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kserve-container/0.log"
Aug 04 18:26:59.749272 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:59.749253    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kube-rbac-proxy/0.log"
Aug 04 18:26:59.760675 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:59.760659    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/agent/0.log"
Aug 04 18:26:59.771515 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:26:59.771499    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/storage-initializer/0.log"
Aug 04 18:27:00.569471 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:00.569448    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kserve-container/0.log"
Aug 04 18:27:00.586629 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:00.586609    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kube-rbac-proxy/0.log"
Aug 04 18:27:00.603433 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:00.603411    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/agent/0.log"
Aug 04 18:27:00.625258 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:00.625243    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/storage-initializer/0.log"
Aug 04 18:27:01.469905 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:01.469883    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kserve-container/0.log"
Aug 04 18:27:01.486811 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:01.486790    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kube-rbac-proxy/0.log"
Aug 04 18:27:01.498890 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:01.498873    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/agent/0.log"
Aug 04 18:27:01.510816 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:01.510795    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/storage-initializer/0.log"
Aug 04 18:27:02.313284 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:02.313253    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kserve-container/0.log"
Aug 04 18:27:02.330852 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:02.330830    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kube-rbac-proxy/0.log"
Aug 04 18:27:02.344990 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:02.344970    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/agent/0.log"
Aug 04 18:27:02.357860 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:02.357842    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/storage-initializer/0.log"
Aug 04 18:27:03.152001 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:03.151975    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kserve-container/0.log"
Aug 04 18:27:03.165222 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:03.165198    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kube-rbac-proxy/0.log"
Aug 04 18:27:03.177886 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:03.177870    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/agent/0.log"
Aug 04 18:27:03.192304 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:03.192279    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/storage-initializer/0.log"
Aug 04 18:27:03.999854 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:03.999830    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Aug 04 18:27:04.005173 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:04.005150    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:27:04.005274 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:04.005262    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:27:04.006708 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:04.006679    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:27:04.006817 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:04.006780    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:27:04.135339 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:04.135304    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kserve-container/0.log"
Aug 04 18:27:04.150656 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:04.150634    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/kube-rbac-proxy/0.log"
Aug 04 18:27:04.163129 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:04.163114    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/agent/0.log"
Aug 04 18:27:04.174894 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:04.174876    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99_412e335e-6bcd-4251-af23-971bac722e3d/storage-initializer/0.log"
Aug 04 18:27:07.041076 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.041053    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:27:07.161061 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.161000    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cqd9\" (UniqueName: \"kubernetes.io/projected/412e335e-6bcd-4251-af23-971bac722e3d-kube-api-access-2cqd9\") pod \"412e335e-6bcd-4251-af23-971bac722e3d\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") "
Aug 04 18:27:07.161061 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.161031    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/412e335e-6bcd-4251-af23-971bac722e3d-kube-rbac-proxy-sar-config\") pod \"412e335e-6bcd-4251-af23-971bac722e3d\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") "
Aug 04 18:27:07.161221 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.161067    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/412e335e-6bcd-4251-af23-971bac722e3d-kserve-provision-location\") pod \"412e335e-6bcd-4251-af23-971bac722e3d\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") "
Aug 04 18:27:07.161221 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.161084    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/412e335e-6bcd-4251-af23-971bac722e3d-cabundle-cert\") pod \"412e335e-6bcd-4251-af23-971bac722e3d\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") "
Aug 04 18:27:07.161221 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.161175    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412e335e-6bcd-4251-af23-971bac722e3d-proxy-tls\") pod \"412e335e-6bcd-4251-af23-971bac722e3d\" (UID: \"412e335e-6bcd-4251-af23-971bac722e3d\") "
Aug 04 18:27:07.161452 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.161424    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412e335e-6bcd-4251-af23-971bac722e3d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "412e335e-6bcd-4251-af23-971bac722e3d" (UID: "412e335e-6bcd-4251-af23-971bac722e3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:07.161549 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.161449    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412e335e-6bcd-4251-af23-971bac722e3d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "412e335e-6bcd-4251-af23-971bac722e3d" (UID: "412e335e-6bcd-4251-af23-971bac722e3d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:27:07.161549 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.161503    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412e335e-6bcd-4251-af23-971bac722e3d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "412e335e-6bcd-4251-af23-971bac722e3d" (UID: "412e335e-6bcd-4251-af23-971bac722e3d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:27:07.162967 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.162944    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412e335e-6bcd-4251-af23-971bac722e3d-kube-api-access-2cqd9" (OuterVolumeSpecName: "kube-api-access-2cqd9") pod "412e335e-6bcd-4251-af23-971bac722e3d" (UID: "412e335e-6bcd-4251-af23-971bac722e3d"). InnerVolumeSpecName "kube-api-access-2cqd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:27:07.163135 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.163114    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/412e335e-6bcd-4251-af23-971bac722e3d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "412e335e-6bcd-4251-af23-971bac722e3d" (UID: "412e335e-6bcd-4251-af23-971bac722e3d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:27:07.262612 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.262580    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/412e335e-6bcd-4251-af23-971bac722e3d-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:27:07.262612 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.262609    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cqd9\" (UniqueName: \"kubernetes.io/projected/412e335e-6bcd-4251-af23-971bac722e3d-kube-api-access-2cqd9\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:27:07.262612 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.262621    2543 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/412e335e-6bcd-4251-af23-971bac722e3d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:27:07.262774 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.262629    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/412e335e-6bcd-4251-af23-971bac722e3d-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:27:07.262774 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.262637    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/412e335e-6bcd-4251-af23-971bac722e3d-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Aug 04 18:27:07.411594 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.411547    2543 generic.go:356] "Generic (PLEG): container finished" podID="412e335e-6bcd-4251-af23-971bac722e3d" containerID="6fb3e8d44e71a2d52573a9ea368373ee91f1800aab5e78b8bc8781347d786ec0" exitCode=0
Aug 04 18:27:07.411673 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.411604    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"
Aug 04 18:27:07.411736 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.411603    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" event={"ID":"412e335e-6bcd-4251-af23-971bac722e3d","Type":"ContainerDied","Data":"6fb3e8d44e71a2d52573a9ea368373ee91f1800aab5e78b8bc8781347d786ec0"}
Aug 04 18:27:07.411789 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.411764    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99" event={"ID":"412e335e-6bcd-4251-af23-971bac722e3d","Type":"ContainerDied","Data":"261430379eebe05309fb6fd1bc73be9f9384fc808b1e881990b5f271e3bde400"}
Aug 04 18:27:07.411839 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.411819    2543 scope.go:117] "RemoveContainer" containerID="6fb3e8d44e71a2d52573a9ea368373ee91f1800aab5e78b8bc8781347d786ec0"
Aug 04 18:27:07.419216 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.419198    2543 scope.go:117] "RemoveContainer" containerID="fedc5d00a246d523c5a9e204259f51815826a9ec060bd6339f6d4b7848c58e8b"
Aug 04 18:27:07.425999 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.425984    2543 scope.go:117] "RemoveContainer" containerID="120933e55ab69bd826f059f4df275edef7c57380c0e71d89c6ee01d9135f83a6"
Aug 04 18:27:07.432286 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.432268    2543 scope.go:117] "RemoveContainer" containerID="0c79d3913c2154422293feafa7f267fd9bb24725112ea0e560b102c4bcdd266a"
Aug 04 18:27:07.436083 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.436063    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"]
Aug 04 18:27:07.439443 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.439423    2543 scope.go:117] "RemoveContainer" containerID="6fb3e8d44e71a2d52573a9ea368373ee91f1800aab5e78b8bc8781347d786ec0"
Aug 04 18:27:07.439713 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:27:07.439692    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb3e8d44e71a2d52573a9ea368373ee91f1800aab5e78b8bc8781347d786ec0\": container with ID starting with 6fb3e8d44e71a2d52573a9ea368373ee91f1800aab5e78b8bc8781347d786ec0 not found: ID does not exist" containerID="6fb3e8d44e71a2d52573a9ea368373ee91f1800aab5e78b8bc8781347d786ec0"
Aug 04 18:27:07.439813 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.439724    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb3e8d44e71a2d52573a9ea368373ee91f1800aab5e78b8bc8781347d786ec0"} err="failed to get container status \"6fb3e8d44e71a2d52573a9ea368373ee91f1800aab5e78b8bc8781347d786ec0\": rpc error: code = NotFound desc = could not find container \"6fb3e8d44e71a2d52573a9ea368373ee91f1800aab5e78b8bc8781347d786ec0\": container with ID starting with 6fb3e8d44e71a2d52573a9ea368373ee91f1800aab5e78b8bc8781347d786ec0 not found: ID does not exist"
Aug 04 18:27:07.439813 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.439748    2543 scope.go:117] "RemoveContainer" containerID="fedc5d00a246d523c5a9e204259f51815826a9ec060bd6339f6d4b7848c58e8b"
Aug 04 18:27:07.440018 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:27:07.439996    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fedc5d00a246d523c5a9e204259f51815826a9ec060bd6339f6d4b7848c58e8b\": container with ID starting with fedc5d00a246d523c5a9e204259f51815826a9ec060bd6339f6d4b7848c58e8b not found: ID does not exist" containerID="fedc5d00a246d523c5a9e204259f51815826a9ec060bd6339f6d4b7848c58e8b"
Aug 04 18:27:07.440059 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.440027    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fedc5d00a246d523c5a9e204259f51815826a9ec060bd6339f6d4b7848c58e8b"} err="failed to get container status \"fedc5d00a246d523c5a9e204259f51815826a9ec060bd6339f6d4b7848c58e8b\": rpc error: code = NotFound desc = could not find container \"fedc5d00a246d523c5a9e204259f51815826a9ec060bd6339f6d4b7848c58e8b\": container with ID starting with fedc5d00a246d523c5a9e204259f51815826a9ec060bd6339f6d4b7848c58e8b not found: ID does not exist"
Aug 04 18:27:07.440059 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.440047    2543 scope.go:117] "RemoveContainer" containerID="120933e55ab69bd826f059f4df275edef7c57380c0e71d89c6ee01d9135f83a6"
Aug 04 18:27:07.440287 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:27:07.440269    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"120933e55ab69bd826f059f4df275edef7c57380c0e71d89c6ee01d9135f83a6\": container with ID starting with 120933e55ab69bd826f059f4df275edef7c57380c0e71d89c6ee01d9135f83a6 not found: ID does not exist" containerID="120933e55ab69bd826f059f4df275edef7c57380c0e71d89c6ee01d9135f83a6"
Aug 04 18:27:07.440381 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.440294    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"120933e55ab69bd826f059f4df275edef7c57380c0e71d89c6ee01d9135f83a6"} err="failed to get container status \"120933e55ab69bd826f059f4df275edef7c57380c0e71d89c6ee01d9135f83a6\": rpc error: code = NotFound desc = could not find container \"120933e55ab69bd826f059f4df275edef7c57380c0e71d89c6ee01d9135f83a6\": container with ID starting with 120933e55ab69bd826f059f4df275edef7c57380c0e71d89c6ee01d9135f83a6 not found: ID does not exist"
Aug 04 18:27:07.440381 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.440347    2543 scope.go:117] "RemoveContainer" containerID="0c79d3913c2154422293feafa7f267fd9bb24725112ea0e560b102c4bcdd266a"
Aug 04 18:27:07.440592 ip-10-0-139-237 kubenswrapper[2543]: E0804 18:27:07.440573    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c79d3913c2154422293feafa7f267fd9bb24725112ea0e560b102c4bcdd266a\": container with ID starting with 0c79d3913c2154422293feafa7f267fd9bb24725112ea0e560b102c4bcdd266a not found: ID does not exist" containerID="0c79d3913c2154422293feafa7f267fd9bb24725112ea0e560b102c4bcdd266a"
Aug 04 18:27:07.440633 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.440597    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c79d3913c2154422293feafa7f267fd9bb24725112ea0e560b102c4bcdd266a"} err="failed to get container status \"0c79d3913c2154422293feafa7f267fd9bb24725112ea0e560b102c4bcdd266a\": rpc error: code = NotFound desc = could not find container \"0c79d3913c2154422293feafa7f267fd9bb24725112ea0e560b102c4bcdd266a\": container with ID starting with 0c79d3913c2154422293feafa7f267fd9bb24725112ea0e560b102c4bcdd266a not found: ID does not exist"
Aug 04 18:27:07.440942 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.440925    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-94ff54bf4-vzx99"]
Aug 04 18:27:07.575692 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:07.575671    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="412e335e-6bcd-4251-af23-971bac722e3d" path="/var/lib/kubelet/pods/412e335e-6bcd-4251-af23-971bac722e3d/volumes"
Aug 04 18:27:08.238797 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.238774    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-flh2v/must-gather-6cvbl"]
Aug 04 18:27:08.239123 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.239030    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="agent"
Aug 04 18:27:08.239123 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.239040    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="agent"
Aug 04 18:27:08.239123 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.239052    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="storage-initializer"
Aug 04 18:27:08.239123 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.239058    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="storage-initializer"
Aug 04 18:27:08.239123 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.239073    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kserve-container"
Aug 04 18:27:08.239123 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.239078    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kserve-container"
Aug 04 18:27:08.239123 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.239087    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kube-rbac-proxy"
Aug 04 18:27:08.239123 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.239092    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kube-rbac-proxy"
Aug 04 18:27:08.239414 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.239148    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="agent"
Aug 04 18:27:08.239414 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.239157    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kserve-container"
Aug 04 18:27:08.239414 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.239165    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="412e335e-6bcd-4251-af23-971bac722e3d" containerName="kube-rbac-proxy"
Aug 04 18:27:08.243535 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.243521    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-flh2v/must-gather-6cvbl"
Aug 04 18:27:08.246186 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.246160    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-flh2v\"/\"openshift-service-ca.crt\""
Aug 04 18:27:08.246298 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.246201    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-flh2v\"/\"kube-root-ca.crt\""
Aug 04 18:27:08.246298 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.246225    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-flh2v\"/\"default-dockercfg-g5r45\""
Aug 04 18:27:08.254434 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.254415    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-flh2v/must-gather-6cvbl"]
Aug 04 18:27:08.369990 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.369962    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/55a8e1eb-e637-4fc5-9db3-e5b3adfcd393-must-gather-output\") pod \"must-gather-6cvbl\" (UID: \"55a8e1eb-e637-4fc5-9db3-e5b3adfcd393\") " pod="openshift-must-gather-flh2v/must-gather-6cvbl"
Aug 04 18:27:08.370090 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.370014    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qq2r\" (UniqueName: \"kubernetes.io/projected/55a8e1eb-e637-4fc5-9db3-e5b3adfcd393-kube-api-access-4qq2r\") pod \"must-gather-6cvbl\" (UID: \"55a8e1eb-e637-4fc5-9db3-e5b3adfcd393\") " pod="openshift-must-gather-flh2v/must-gather-6cvbl"
Aug 04 18:27:08.471259 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.471238    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/55a8e1eb-e637-4fc5-9db3-e5b3adfcd393-must-gather-output\") pod \"must-gather-6cvbl\" (UID: \"55a8e1eb-e637-4fc5-9db3-e5b3adfcd393\") " pod="openshift-must-gather-flh2v/must-gather-6cvbl"
Aug 04 18:27:08.471366 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.471275    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qq2r\" (UniqueName: \"kubernetes.io/projected/55a8e1eb-e637-4fc5-9db3-e5b3adfcd393-kube-api-access-4qq2r\") pod \"must-gather-6cvbl\" (UID: \"55a8e1eb-e637-4fc5-9db3-e5b3adfcd393\") " pod="openshift-must-gather-flh2v/must-gather-6cvbl"
Aug 04 18:27:08.471539 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.471523    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/55a8e1eb-e637-4fc5-9db3-e5b3adfcd393-must-gather-output\") pod \"must-gather-6cvbl\" (UID: \"55a8e1eb-e637-4fc5-9db3-e5b3adfcd393\") " pod="openshift-must-gather-flh2v/must-gather-6cvbl"
Aug 04 18:27:08.480285 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.480268    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qq2r\" (UniqueName: \"kubernetes.io/projected/55a8e1eb-e637-4fc5-9db3-e5b3adfcd393-kube-api-access-4qq2r\") pod \"must-gather-6cvbl\" (UID: \"55a8e1eb-e637-4fc5-9db3-e5b3adfcd393\") " pod="openshift-must-gather-flh2v/must-gather-6cvbl"
Aug 04 18:27:08.552333 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.552272    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-flh2v/must-gather-6cvbl"
Aug 04 18:27:08.675199 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.675173    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-flh2v/must-gather-6cvbl"]
Aug 04 18:27:08.677457 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:27:08.677431    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55a8e1eb_e637_4fc5_9db3_e5b3adfcd393.slice/crio-36d572c21d6ae2d7b3ac0b3cd224e440da124e9addb6d40a7d7e0e23b718a3bf WatchSource:0}: Error finding container 36d572c21d6ae2d7b3ac0b3cd224e440da124e9addb6d40a7d7e0e23b718a3bf: Status 404 returned error can't find the container with id 36d572c21d6ae2d7b3ac0b3cd224e440da124e9addb6d40a7d7e0e23b718a3bf
Aug 04 18:27:08.679185 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:08.679168    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:27:09.421465 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:09.421431    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-flh2v/must-gather-6cvbl" event={"ID":"55a8e1eb-e637-4fc5-9db3-e5b3adfcd393","Type":"ContainerStarted","Data":"36d572c21d6ae2d7b3ac0b3cd224e440da124e9addb6d40a7d7e0e23b718a3bf"}
Aug 04 18:27:10.427902 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:10.427869    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-flh2v/must-gather-6cvbl" event={"ID":"55a8e1eb-e637-4fc5-9db3-e5b3adfcd393","Type":"ContainerStarted","Data":"c99251490ca5e6dd911c55bad7193add32652c2f24093ca4c4deb6891e30954b"}
Aug 04 18:27:10.428285 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:10.427911    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-flh2v/must-gather-6cvbl" event={"ID":"55a8e1eb-e637-4fc5-9db3-e5b3adfcd393","Type":"ContainerStarted","Data":"e49773e87cc483179c93a32c344d70e6c2745a0022606999fcad5728f4f03558"}
Aug 04 18:27:10.444878 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:10.444828    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-flh2v/must-gather-6cvbl" podStartSLOduration=1.665836472 podStartE2EDuration="2.444813426s" totalImagesPullingTime="778.976954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:27:10.443669207 +0000 UTC m=+1633.414313553" watchObservedRunningTime="2026-08-04 18:27:10.444813426 +0000 UTC m=+1633.415457765"
Aug 04 18:27:10.912866 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:10.912819    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gsbtv_59896da8-3715-4ab5-b4b7-cbce0026cc98/global-pull-secret-syncer/0.log"
Aug 04 18:27:11.084086 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:11.084056    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bcz4n_9aeabbb3-4723-4845-b296-bc37aa059605/konnectivity-agent/0.log"
Aug 04 18:27:11.158115 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:11.158087    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-237.ec2.internal_28c305bdf94bdf64315ff2bce273b65b/haproxy/0.log"
Aug 04 18:27:15.374400 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:15.374371    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-5vmkh_fc58f3af-bbf6-4904-a741-809c9dbeac97/kube-state-metrics/0.log"
Aug 04 18:27:15.399519 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:15.399483    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-5vmkh_fc58f3af-bbf6-4904-a741-809c9dbeac97/kube-rbac-proxy-main/0.log"
Aug 04 18:27:15.423124 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:15.423099    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-5vmkh_fc58f3af-bbf6-4904-a741-809c9dbeac97/kube-rbac-proxy-self/0.log"
Aug 04 18:27:15.464603 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:15.464576    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7cc78b8f7d-96w89_bf48811a-ab2a-4931-bdb3-bd99d3378bad/metrics-server/0.log"
Aug 04 18:27:15.721956 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:15.721918    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nv7dq_e591a246-2dbf-4e5a-bf50-14b1099c86ab/node-exporter/0.log"
Aug 04 18:27:15.751721 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:15.751661    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nv7dq_e591a246-2dbf-4e5a-bf50-14b1099c86ab/kube-rbac-proxy/0.log"
Aug 04 18:27:15.779436 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:15.779414    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nv7dq_e591a246-2dbf-4e5a-bf50-14b1099c86ab/init-textfile/0.log"
Aug 04 18:27:15.912481 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:15.912445    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0c3ff52-1d60-480c-a73d-f074d4eee205/prometheus/0.log"
Aug 04 18:27:15.951081 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:15.951054    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0c3ff52-1d60-480c-a73d-f074d4eee205/config-reloader/0.log"
Aug 04 18:27:15.980735 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:15.980617    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0c3ff52-1d60-480c-a73d-f074d4eee205/thanos-sidecar/0.log"
Aug 04 18:27:16.010325 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:16.010288    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0c3ff52-1d60-480c-a73d-f074d4eee205/kube-rbac-proxy-web/0.log"
Aug 04 18:27:16.038655 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:16.038632    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0c3ff52-1d60-480c-a73d-f074d4eee205/kube-rbac-proxy/0.log"
Aug 04 18:27:16.078657 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:16.078621    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0c3ff52-1d60-480c-a73d-f074d4eee205/kube-rbac-proxy-thanos/0.log"
Aug 04 18:27:16.106800 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:16.106765    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0c3ff52-1d60-480c-a73d-f074d4eee205/init-config-reloader/0.log"
Aug 04 18:27:16.154704 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:16.154671    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-n25fm_ef679628-6b96-4bce-9ee7-736b32f4d2f5/prometheus-operator/0.log"
Aug 04 18:27:16.186092 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:16.186031    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-n25fm_ef679628-6b96-4bce-9ee7-736b32f4d2f5/kube-rbac-proxy/0.log"
Aug 04 18:27:16.223334 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:16.221863    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-k9ktl_726c7c38-d14e-4db0-b34d-d8ad5bb965b1/prometheus-operator-admission-webhook/0.log"
Aug 04 18:27:18.253670 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.253640    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/1.log"
Aug 04 18:27:18.260094 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.260071    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wftvq_57c7e8ed-7892-40d4-aebe-ccd866618ccc/console-operator/2.log"
Aug 04 18:27:18.634266 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.634233    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"]
Aug 04 18:27:18.639808 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.639781    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.651278 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.651253    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"]
Aug 04 18:27:18.665798 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.665765    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31bf01ab-c865-47fe-a1c8-75240ec0177a-sys\") pod \"perf-node-gather-daemonset-bhx74\" (UID: \"31bf01ab-c865-47fe-a1c8-75240ec0177a\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.665943 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.665798    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzc4l\" (UniqueName: \"kubernetes.io/projected/31bf01ab-c865-47fe-a1c8-75240ec0177a-kube-api-access-gzc4l\") pod \"perf-node-gather-daemonset-bhx74\" (UID: \"31bf01ab-c865-47fe-a1c8-75240ec0177a\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.665943 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.665834    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31bf01ab-c865-47fe-a1c8-75240ec0177a-podres\") pod \"perf-node-gather-daemonset-bhx74\" (UID: \"31bf01ab-c865-47fe-a1c8-75240ec0177a\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.665943 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.665907    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31bf01ab-c865-47fe-a1c8-75240ec0177a-proc\") pod \"perf-node-gather-daemonset-bhx74\" (UID: \"31bf01ab-c865-47fe-a1c8-75240ec0177a\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.665943 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.665933    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31bf01ab-c865-47fe-a1c8-75240ec0177a-lib-modules\") pod \"perf-node-gather-daemonset-bhx74\" (UID: \"31bf01ab-c865-47fe-a1c8-75240ec0177a\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.738993 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.738966    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-q6rn5_3aa47b21-f214-40ef-a37d-29d68ba50805/download-server/0.log"
Aug 04 18:27:18.766632 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.766592    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31bf01ab-c865-47fe-a1c8-75240ec0177a-sys\") pod \"perf-node-gather-daemonset-bhx74\" (UID: \"31bf01ab-c865-47fe-a1c8-75240ec0177a\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.766632 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.766631    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzc4l\" (UniqueName: \"kubernetes.io/projected/31bf01ab-c865-47fe-a1c8-75240ec0177a-kube-api-access-gzc4l\") pod \"perf-node-gather-daemonset-bhx74\" (UID: \"31bf01ab-c865-47fe-a1c8-75240ec0177a\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.766850 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.766669    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31bf01ab-c865-47fe-a1c8-75240ec0177a-podres\") pod \"perf-node-gather-daemonset-bhx74\" (UID: \"31bf01ab-c865-47fe-a1c8-75240ec0177a\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.766850 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.766705    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31bf01ab-c865-47fe-a1c8-75240ec0177a-proc\") pod \"perf-node-gather-daemonset-bhx74\" (UID: \"31bf01ab-c865-47fe-a1c8-75240ec0177a\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.766850 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.766705    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31bf01ab-c865-47fe-a1c8-75240ec0177a-sys\") pod \"perf-node-gather-daemonset-bhx74\" (UID: \"31bf01ab-c865-47fe-a1c8-75240ec0177a\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.766850 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.766738    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31bf01ab-c865-47fe-a1c8-75240ec0177a-lib-modules\") pod \"perf-node-gather-daemonset-bhx74\" (UID: \"31bf01ab-c865-47fe-a1c8-75240ec0177a\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.766850 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.766829    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31bf01ab-c865-47fe-a1c8-75240ec0177a-podres\") pod \"perf-node-gather-daemonset-bhx74\" (UID: \"31bf01ab-c865-47fe-a1c8-75240ec0177a\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.767068 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.766878    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31bf01ab-c865-47fe-a1c8-75240ec0177a-proc\") pod \"perf-node-gather-daemonset-bhx74\" (UID: \"31bf01ab-c865-47fe-a1c8-75240ec0177a\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.767068 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.766893    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31bf01ab-c865-47fe-a1c8-75240ec0177a-lib-modules\") pod \"perf-node-gather-daemonset-bhx74\" (UID: \"31bf01ab-c865-47fe-a1c8-75240ec0177a\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.776830 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.776803    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzc4l\" (UniqueName: \"kubernetes.io/projected/31bf01ab-c865-47fe-a1c8-75240ec0177a-kube-api-access-gzc4l\") pod \"perf-node-gather-daemonset-bhx74\" (UID: \"31bf01ab-c865-47fe-a1c8-75240ec0177a\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:18.952529 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:18.952455    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:19.109746 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:19.109708    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"]
Aug 04 18:27:19.114783 ip-10-0-139-237 kubenswrapper[2543]: W0804 18:27:19.114757    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod31bf01ab_c865_47fe_a1c8_75240ec0177a.slice/crio-8786c11a48cc18fa8c2ce3de7f63c7275fe6fcd07e1e9ed7c2e6704acf6dd6ec WatchSource:0}: Error finding container 8786c11a48cc18fa8c2ce3de7f63c7275fe6fcd07e1e9ed7c2e6704acf6dd6ec: Status 404 returned error can't find the container with id 8786c11a48cc18fa8c2ce3de7f63c7275fe6fcd07e1e9ed7c2e6704acf6dd6ec
Aug 04 18:27:19.479984 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:19.479948    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74" event={"ID":"31bf01ab-c865-47fe-a1c8-75240ec0177a","Type":"ContainerStarted","Data":"7d9c68ed4e239fca519f32cf26b7bdc0a5f9f70b2e73736ff0356684a47e4c9d"}
Aug 04 18:27:19.480474 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:19.480014    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74" event={"ID":"31bf01ab-c865-47fe-a1c8-75240ec0177a","Type":"ContainerStarted","Data":"8786c11a48cc18fa8c2ce3de7f63c7275fe6fcd07e1e9ed7c2e6704acf6dd6ec"}
Aug 04 18:27:19.480474 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:19.480070    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:19.498638 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:19.498582    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74" podStartSLOduration=1.498567147 podStartE2EDuration="1.498567147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:27:19.497551459 +0000 UTC m=+1642.468195798" watchObservedRunningTime="2026-08-04 18:27:19.498567147 +0000 UTC m=+1642.469211478"
Aug 04 18:27:20.285022 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:20.284994    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r9ntf_62c75b76-1fdb-4555-a35e-78d0d361c064/dns/0.log"
Aug 04 18:27:20.306969 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:20.306950    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r9ntf_62c75b76-1fdb-4555-a35e-78d0d361c064/kube-rbac-proxy/0.log"
Aug 04 18:27:20.353774 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:20.353756    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p52sc_d05e7582-030c-4b50-b5ce-90e4c81c1f1e/dns-node-resolver/0.log"
Aug 04 18:27:20.827180 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:20.827155    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z8dv8_dea9d566-e866-4409-ad9a-108474118ce2/node-ca/0.log"
Aug 04 18:27:21.990244 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:21.990212    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fk7ll_68ff79ec-393a-456b-9041-3389e96656fc/serve-healthcheck-canary/0.log"
Aug 04 18:27:22.415888 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:22.415858    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-cvxwr_7b6607f0-aea9-4a84-a327-6794dd2b7dd5/insights-operator/0.log"
Aug 04 18:27:22.416057 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:22.415929    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-cvxwr_7b6607f0-aea9-4a84-a327-6794dd2b7dd5/insights-operator/1.log"
Aug 04 18:27:22.583959 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:22.583932    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w9k9t_2c1974c9-ee39-4f0d-bcae-c749b4a26519/kube-rbac-proxy/0.log"
Aug 04 18:27:22.605938 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:22.605919    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w9k9t_2c1974c9-ee39-4f0d-bcae-c749b4a26519/exporter/0.log"
Aug 04 18:27:22.629383 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:22.629364    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w9k9t_2c1974c9-ee39-4f0d-bcae-c749b4a26519/extractor/0.log"
Aug 04 18:27:24.990577 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:24.990552    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-8588677cfb-rf2zl_b8ff864e-167e-48eb-a074-3253b1581e79/manager/0.log"
Aug 04 18:27:25.022482 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:25.022459    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2xwsw_f9a69cf6-a3e9-431a-bcdc-9373c002910e/s3-init/0.log"
Aug 04 18:27:25.041425 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:25.041400    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2xwsw_f9a69cf6-a3e9-431a-bcdc-9373c002910e/download-hf-model/0.log"
Aug 04 18:27:25.493784 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:25.493757    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-bhx74"
Aug 04 18:27:30.654393 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:30.654368    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zthc_d3889c85-d79f-4629-94bb-c3fe1c1471ff/kube-multus-additional-cni-plugins/0.log"
Aug 04 18:27:30.675927 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:30.675869    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zthc_d3889c85-d79f-4629-94bb-c3fe1c1471ff/egress-router-binary-copy/0.log"
Aug 04 18:27:30.698972 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:30.698955    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zthc_d3889c85-d79f-4629-94bb-c3fe1c1471ff/cni-plugins/0.log"
Aug 04 18:27:30.720724 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:30.720708    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zthc_d3889c85-d79f-4629-94bb-c3fe1c1471ff/bond-cni-plugin/0.log"
Aug 04 18:27:30.749637 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:30.749614    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zthc_d3889c85-d79f-4629-94bb-c3fe1c1471ff/routeoverride-cni/0.log"
Aug 04 18:27:30.773627 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:30.773584    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zthc_d3889c85-d79f-4629-94bb-c3fe1c1471ff/whereabouts-cni-bincopy/0.log"
Aug 04 18:27:30.795995 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:30.795974    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zthc_d3889c85-d79f-4629-94bb-c3fe1c1471ff/whereabouts-cni/0.log"
Aug 04 18:27:31.180856 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:31.180834    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-p4xjn_fa71b16a-485e-483f-b079-3c00e21b44e6/kube-multus/0.log"
Aug 04 18:27:31.295914 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:31.295868    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r27tq_4fea01c7-a50c-40f7-a628-9745002f5158/network-metrics-daemon/0.log"
Aug 04 18:27:31.316493 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:31.316469    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r27tq_4fea01c7-a50c-40f7-a628-9745002f5158/kube-rbac-proxy/0.log"
Aug 04 18:27:32.512153 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:32.512110    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkvl9_71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3/ovn-controller/0.log"
Aug 04 18:27:32.531199 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:32.531170    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkvl9_71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3/ovn-acl-logging/0.log"
Aug 04 18:27:32.563616 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:32.563594    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkvl9_71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3/kube-rbac-proxy-node/0.log"
Aug 04 18:27:32.588009 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:32.587989    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkvl9_71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3/kube-rbac-proxy-ovn-metrics/0.log"
Aug 04 18:27:32.607801 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:32.607783    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkvl9_71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3/northd/0.log"
Aug 04 18:27:32.631020 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:32.631006    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkvl9_71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3/nbdb/0.log"
Aug 04 18:27:32.652782 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:32.652729    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkvl9_71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3/sbdb/0.log"
Aug 04 18:27:32.695491 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:32.695474    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkvl9_71d9fb85-07cf-4f2a-9e77-04eaa8d80ce3/ovnkube-controller/0.log"
Aug 04 18:27:33.704123 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:33.704094    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-65bdbbff45-2txjf_966b1943-a685-4cd3-8f86-3bb210670312/check-endpoints/0.log"
Aug 04 18:27:33.726917 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:33.726893    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-jzdgf_d69a269d-b7ef-4618-aafb-475fc3de7ab2/network-check-target-container/0.log"
Aug 04 18:27:34.717061 ip-10-0-139-237 kubenswrapper[2543]: I0804 18:27:34.717028    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-qnwcg_3e118332-1ad2-42d0-b601-8c566c538432/iptables-alerter/0.log"