Jul 09 17:39:14.531819 ip-10-0-143-142 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 17:39:14.882871 ip-10-0-143-142 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 17:39:14.882871 ip-10-0-143-142 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 17:39:14.882871 ip-10-0-143-142 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 17:39:14.882871 ip-10-0-143-142 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 17:39:14.882871 ip-10-0-143-142 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 17:39:14.882871 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.842046    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 17:39:14.882871 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847313    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:14.882871 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847331    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:14.882871 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847338    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:14.882871 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847348    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:14.882871 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847352    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:14.882871 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847356    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:14.882871 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847360    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:14.882871 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847364    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847369    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847373    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847377    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847383    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847387    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847391    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847397    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847407    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847412    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847417    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847422    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847426    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847430    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847434    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847438    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847443    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847448    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847452    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:14.997301 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847456    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:14.907294 ip-10-0-143-142 systemd[1]: Started Kubernetes Kubelet.
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847461    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847471    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847475    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847482    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847487    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847492    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847497    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847502    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847507    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847511    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847515    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847534    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847539    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847548    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847552    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847556    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847560    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847565    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:14.998567 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847569    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847573    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847578    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847582    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847586    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847591    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847595    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847599    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847618    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847649    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847653    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847656    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847659    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847678    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847682    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847799    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847803    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847806    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847809    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847813    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:14.999361 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847815    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847818    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847821    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847824    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847827    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847830    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847832    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847835    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847838    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847840    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847845    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847848    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847851    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847853    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847858    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847862    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847865    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847868    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847871    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847874    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:15.001379 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847876    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847879    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847882    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847885    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847887    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847890    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847894    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847898    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847901    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847904    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847907    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847910    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847912    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.847929    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849310    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849319    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849322    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849326    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849329    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:15.002214 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849332    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849335    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849338    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849340    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849343    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849345    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849348    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849350    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849353    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849356    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849358    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849361    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849363    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849366    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849370    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849373    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849377    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849380    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849383    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849386    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:15.003403 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849389    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849392    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849395    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849398    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849401    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849404    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849406    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849409    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849411    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849414    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849417    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849420    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849423    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849425    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849428    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849431    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849434    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849436    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849438    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849441    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:15.004226 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849443    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849446    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849448    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849451    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849454    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849456    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849459    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849461    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849464    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849466    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849469    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849472    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849476    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849479    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849485    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849488    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849490    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849493    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849496    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849498    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:15.005427 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849501    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849503    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849506    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849508    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849512    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849515    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849534    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849538    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849540    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849543    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849546    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849549    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849551    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849554    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849556    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849559    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849561    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849564    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849575    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:15.007464 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849578    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849580    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849583    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849585    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849588    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849591    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849594    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849597    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849600    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849603    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849605    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849608    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849610    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849613    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.849615    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850007    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850011    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850015    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850018    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850020    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:15.008145 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850023    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850025    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850028    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850031    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850034    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850036    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850039    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850042    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850044    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850047    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850049    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850053    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850055    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850058    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850060    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850063    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850066    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850069    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850072    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:15.008824 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850078    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850081    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850084    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850086    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850089    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850091    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850094    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850096    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850099    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850102    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850104    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850107    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850109    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850112    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850114    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850117    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850119    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850122    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850125    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:15.009393 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850129    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850131    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850135    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850138    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850141    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850144    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850147    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850150    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850152    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850155    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850159    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850161    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850164    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850169    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850171    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850174    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850176    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850178    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850181    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850183    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:15.009965 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850186    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850188    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850190    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850193    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850196    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850198    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850201    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850203    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850205    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850208    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850210    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850213    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850215    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850218    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850220    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850223    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850225    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850227    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850230    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850235    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:15.010617 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850238    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850241    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850245    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850248    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850251    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850255    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850258    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850260    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850263    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850265    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850268    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850270    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850273    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850276    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850278    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850281    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850352    2573 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850359    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850365    2573 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850370    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850374    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 17:39:15.011208 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850378    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850382    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850386    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850389    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850393    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850396    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850399    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850402    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850406    2573 flags.go:64] FLAG: --cgroup-root=""
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850409    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850412    2573 flags.go:64] FLAG: --client-ca-file=""
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850415    2573 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850418    2573 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850424    2573 flags.go:64] FLAG: --cluster-domain=""
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850428    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850431    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850435    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850438    2573 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850442    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850445    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850448    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850451    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850454    2573 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850457    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 17:39:15.011835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850459    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850463    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850465    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850469    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850472    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850475    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850478    2573 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850481    2573 flags.go:64] FLAG: --enable-server="true"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850484    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850488    2573 flags.go:64] FLAG: --event-burst="100"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850491    2573 flags.go:64] FLAG: --event-qps="50"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850494    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850497    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850500    2573 flags.go:64] FLAG: --eviction-hard=""
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850503    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850506    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850509    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850512    2573 flags.go:64] FLAG: --eviction-soft=""
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850515    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850533    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850541    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850546    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850549    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850552    2573 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850556    2573 flags.go:64] FLAG: --feature-gates=""
Jul 09 17:39:15.012607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850560    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850563    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850566    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850569    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850572    2573 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850575    2573 flags.go:64] FLAG: --help="false"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850578    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-143-142.ec2.internal"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850581    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850584    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850587    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850590    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850593    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850596    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850599    2573 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850602    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850605    2573 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850607    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850610    2573 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850613    2573 flags.go:64] FLAG: --kube-reserved=""
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850616    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850619    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850622    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850625    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850628    2573 flags.go:64] FLAG: --lock-file=""
Jul 09 17:39:15.013379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850630    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850633    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850636    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850641    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850647    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850650    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850653    2573 flags.go:64] FLAG: --logging-format="text"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850657    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850660    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850663    2573 flags.go:64] FLAG: --manifest-url=""
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850666    2573 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850670    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850673    2573 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850677    2573 flags.go:64] FLAG: --max-pods="110"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850681    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850684    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850687    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850689    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850693    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850695    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850698    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850705    2573 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850708    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850712    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 17:39:15.014114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850715    2573 flags.go:64] FLAG: --pod-cidr=""
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850718    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850723    2573 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850726    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850730    2573 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850733    2573 flags.go:64] FLAG: --port="10250"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850736    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850739    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0094d2741b150ce54"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850742    2573 flags.go:64] FLAG: --qos-reserved=""
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850745    2573 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850747    2573 flags.go:64] FLAG: --register-node="true"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850751    2573 flags.go:64] FLAG: --register-with-taints=""
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850756    2573 flags.go:64] FLAG: --registry-burst="10"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850760    2573 flags.go:64] FLAG: --registry-qps="5"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850763    2573 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850766    2573 flags.go:64] FLAG: --reserved-memory=""
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850771    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850774    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850777    2573 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850780    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850783    2573 flags.go:64] FLAG: --runonce="false"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850786    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850789    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850792    2573 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850795    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 17:39:15.014842 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850797    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850800    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850803    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850806    2573 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850809    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850812    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850815    2573 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850818    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850821    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850824    2573 flags.go:64] FLAG: --system-cgroups=""
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850827    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850832    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850835    2573 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850838    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850842    2573 flags.go:64] FLAG: --tls-min-version=""
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850845    2573 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850848    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850851    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850853    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850858    2573 flags.go:64] FLAG: --v="2"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850861    2573 flags.go:64] FLAG: --version="false"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850866    2573 flags.go:64] FLAG: --vmodule=""
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850870    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.850874    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.850997    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:15.015614 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851001    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851005    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851008    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851010    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851013    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851016    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851019    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851021    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851024    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851026    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851029    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851031    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851034    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851036    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851040    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851044    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851047    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851050    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851053    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:15.016360 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851056    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851058    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851061    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851063    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851066    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851068    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851071    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851075    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851078    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851080    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851083    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851088    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851091    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851093    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851096    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851099    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851101    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851103    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851106    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851108    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:15.016994 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851111    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851113    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851116    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851118    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851121    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851123    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851126    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851128    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851131    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851133    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851136    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851139    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851141    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851144    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851146    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851149    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851151    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851154    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851156    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851160    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:15.017606 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851163    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851166    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851168    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851173    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851176    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851178    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851181    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851184    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851186    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851189    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851191    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851194    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851196    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851199    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851201    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851203    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851206    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851208    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851211    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:15.018215 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851213    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851216    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851219    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851221    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851224    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851226    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851229    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851231    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851233    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851236    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851238    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851241    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851246    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851250    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851252    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851256    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851261    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851265    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851268    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:15.018793 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.851271    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:15.019359 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.851775    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 17:39:15.019359 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.858176    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 17:39:15.019359 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.858190    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 17:39:15.019359 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.858906    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 17:39:15.019359 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.858913    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 17:39:15.019359 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.858968    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:15.019359 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.858973    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:15.019359 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.858977    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:15.019359 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.858980    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:15.019359 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.858983    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:15.019359 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.858988    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:15.019359 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.858991    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:15.019359 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.858995    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:15.019359 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.858998    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859001    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859004    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859007    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859009    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859012    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859015    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859017    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859020    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859023    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859025    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859028    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859031    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859033    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859036    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859038    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859044    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859047    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859050    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:15.019813 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859052    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859055    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859057    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859060    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859063    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859065    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859068    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859071    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859074    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859076    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859079    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859081    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859084    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859086    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859089    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859091    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859094    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859096    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859099    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859101    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:15.020375 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859104    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859107    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859110    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859112    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859115    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859117    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859120    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859122    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859125    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859127    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859131    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859133    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859136    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859138    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859141    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859145    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859149    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859152    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859156    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859159    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:15.020992 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859161    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859164    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859166    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859169    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859172    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859174    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859177    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859180    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859182    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859185    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859187    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859190    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859192    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859195    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859197    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859201    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859204    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859207    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859210    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:15.021605 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859213    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:15.022188 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859216    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:15.022188 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859218    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:15.022188 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859221    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:15.022188 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859224    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:15.022188 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859227    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:15.022188 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859229    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:15.022188 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859234    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:15.022188 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859237    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:15.022188 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859240    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:15.022188 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859243    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:15.022188 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859246    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:15.022188 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859248    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:15.022188 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.859253    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 17:39:15.022188 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859364    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:15.022188 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859369    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859371    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859375    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859378    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859381    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859384    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859386    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859389    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859391    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859394    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859396    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859399    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859401    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859403    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859406    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859409    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859411    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859414    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859416    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859419    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:15.022661 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859421    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859423    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859426    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859429    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859431    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859436    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859438    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859441    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859445    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859447    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859450    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859452    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859455    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859458    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859462    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859465    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859468    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859471    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859474    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:15.023261 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859477    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859479    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859482    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859484    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859487    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859489    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859492    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859494    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859497    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859499    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859501    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859504    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859507    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859509    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859511    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859514    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859517    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859533    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859538    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859541    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:15.023850 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859543    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859547    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859550    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859552    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859555    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859558    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859560    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859563    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859566    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859570    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859573    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859576    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859579    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859582    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859584    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859587    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859590    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859592    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859595    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:15.024445 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859597    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859600    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859602    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859605    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859608    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859610    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859613    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859615    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859618    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859620    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859624    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859627    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859630    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859633    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859637    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859639    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859642    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859644    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859647    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:15.025048 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:14.859649    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:15.025630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.859654    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 17:39:15.025630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.859749    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 17:39:15.025630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.861752    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 17:39:15.025630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.862883    2573 server.go:1019] "Starting client certificate rotation"
Jul 09 17:39:15.025630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.862976    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 17:39:15.025630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.863016    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 17:39:15.025630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.864921    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 17:39:15.025630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.866405    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 17:39:15.025630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.873750    2573 log.go:25] "Validated CRI v1 runtime API"
Jul 09 17:39:15.025630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.878152    2573 log.go:25] "Validated CRI v1 image API"
Jul 09 17:39:15.025630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.878915    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 17:39:15.025630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.880837    2573 fs.go:135] Filesystem UUIDs: map[0ee89873-198c-45aa-a827-408deceda055:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ec53940a-cdec-4c3c-9421-764710dbc5dd:/dev/nvme0n1p4]
Jul 09 17:39:15.026006 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.880850    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 17:39:15.026045 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.886178    2573 manager.go:216] Machine: {Timestamp:2026-07-09 17:39:14.88436488 +0000 UTC m=+0.256090610 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101850 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fecb584c71fed81e33084471f5f88 SystemUUID:ec2fecb5-84c7-1fed-81e3-3084471f5f88 BootID:037c6aba-9a3d-4d09-92fa-c017b163bf73 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3f:b1:e3:ec:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3f:b1:e3:ec:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:b7:87:d0:93:40 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 17:39:15.026045 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.886278    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 17:39:15.026045 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.886357    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 17:39:15.026045 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.889085    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 17:39:15.026045 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.889111    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-142.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 17:39:15.026045 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.889762    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 17:39:15.026045 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.889771    2573 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.889798    2573 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.889808    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.890407    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.891235    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.894999    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.895114    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.895140    2573 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.895149    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.895159    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.895168    2573 kubelet.go:398] "Adding apiserver pod source"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.895177    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.895531    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.895552    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.897800    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.899213    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.899240    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.899776    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.899790    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.899798    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.899805    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.899815    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.899823    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.899833    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.899839    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.899846    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.899851    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.899860    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.899865    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.901020    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.901032    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.904177    2573 server.go:1268] "Started kubelet"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.904260    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.904306    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.904328    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:14.907381    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.907867    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.907907    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-142.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:14.908279    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 17:39:15.026300 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.909900    2573 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:14.911835    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-142.ec2.internal.18c0afcebcc95341  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-142.ec2.internal,UID:ip-10-0-143-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-142.ec2.internal,},FirstTimestamp:2026-07-09 17:39:14.904150849 +0000 UTC m=+0.275876559,LastTimestamp:2026-07-09 17:39:14.904150849 +0000 UTC m=+0.275876559,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-142.ec2.internal,}"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.913475    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.913482    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:14.913681    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:14.913784    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.914266    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.914279    2573 factory.go:55] Registering systemd factory
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.914290    2573 factory.go:223] Registration of the systemd container factory successfully
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.914366    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.914368    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.914392    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.914474    2573 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.914480    2573 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.914500    2573 factory.go:153] Registering CRI-O factory
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.914514    2573 factory.go:223] Registration of the crio container factory successfully
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.914551    2573 factory.go:103] Registering Raw factory
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.914563    2573 manager.go:1196] Started watching for new ooms in manager
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.915144    2573 manager.go:349] Starting recovery of all containers
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:14.915405    2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:14.915494    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.925578    2573 manager.go:354] Recovery completed
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.931092    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.933098    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.933136    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.933149    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.933734    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.933747    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.933748    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.933834    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.935945    2573 policy_none.go:49] "None policy: Start"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.935958    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.935967    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.937886    2573 policy_none.go:47] "Start"
Jul 09 17:39:15.027498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.944288    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mjnpn"
Jul 09 17:39:15.028627 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:14.950604    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mjnpn"
Jul 09 17:39:15.028627 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:15.014518    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 09 17:39:15.039883 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:15.039859    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:15.039977 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:15.039896    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:15.039977 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:15.039907    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:15.060092 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:15.060070    2573 manager.go:337] "Starting Device Plugin manager"
Jul 09 17:39:15.060206 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:15.060112    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 17:39:15.060206 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:15.060125    2573 server.go:87] "Starting device plugin registration server"
Jul 09 17:39:15.060415 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:15.060396    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 17:39:15.060572 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:15.060414    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 17:39:15.060632 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:15.060581    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 17:39:15.060678 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:15.060671    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 17:39:15.060721 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:15.060696    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 17:39:15.061270 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:15.061235    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 17:39:15.061352 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:15.061291    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 09 17:39:15.121605 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:15.121584    2573 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-142.ec2.internal\" not found" node="ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.518904 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.518875    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:16.520037 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.520015    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:16.520154 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.520053    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:16.520154 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.520065    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:16.520154 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.520093    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.520687 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.520666    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 17:39:16.521794 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.521777    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 17:39:16.521865 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.521823    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 17:39:16.521865 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.521854    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 17:39:16.521927 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:16.521891    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 17:39:16.524016 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.523996    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:16.529474 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.529459    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.529560 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:16.529480    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-142.ec2.internal\": node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 09 17:39:16.579067 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:16.579046    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 09 17:39:16.622142 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.622074    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal"]
Jul 09 17:39:16.622210 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.622165    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:16.623110 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.623093    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:16.623211 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.623128    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:16.623211 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.623141    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:16.625623 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.625607    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:16.625763 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.625741    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.625864 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.625773    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:16.626321 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.626303    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:16.626382 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.626331    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:16.626382 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.626341    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:16.626458 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.626309    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:16.626458 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.626408    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:16.626458 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.626418    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:16.628749 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.628735    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.628798 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.628762    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:16.629410 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.629392    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:16.629489 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.629416    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:16.629489 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.629429    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:16.649289 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:16.649271    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-142.ec2.internal\" not found" node="ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.653501 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:16.653487    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-142.ec2.internal\" not found" node="ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.679992 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:16.679972    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 09 17:39:16.773730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.773677    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/343c2c99cf08d1b78e530233805856b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal\" (UID: \"343c2c99cf08d1b78e530233805856b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.773730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.773737    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/343c2c99cf08d1b78e530233805856b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal\" (UID: \"343c2c99cf08d1b78e530233805856b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.773931 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.773762    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b3b8930aa1846297e408b672c61a6ab-config\") pod \"kube-apiserver-proxy-ip-10-0-143-142.ec2.internal\" (UID: \"0b3b8930aa1846297e408b672c61a6ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.780788 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:16.780764    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 09 17:39:16.874105 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.874044    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/343c2c99cf08d1b78e530233805856b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal\" (UID: \"343c2c99cf08d1b78e530233805856b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.874105 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.874077    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/343c2c99cf08d1b78e530233805856b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal\" (UID: \"343c2c99cf08d1b78e530233805856b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.874105 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.874094    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b3b8930aa1846297e408b672c61a6ab-config\") pod \"kube-apiserver-proxy-ip-10-0-143-142.ec2.internal\" (UID: \"0b3b8930aa1846297e408b672c61a6ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.874254 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.874140    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b3b8930aa1846297e408b672c61a6ab-config\") pod \"kube-apiserver-proxy-ip-10-0-143-142.ec2.internal\" (UID: \"0b3b8930aa1846297e408b672c61a6ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.874254 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.874155    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/343c2c99cf08d1b78e530233805856b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal\" (UID: \"343c2c99cf08d1b78e530233805856b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.874254 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.874155    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/343c2c99cf08d1b78e530233805856b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal\" (UID: \"343c2c99cf08d1b78e530233805856b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.881156 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:16.881141    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 09 17:39:16.952333 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.952315    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.955834 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:16.955818    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal"
Jul 09 17:39:16.981894 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:16.981873    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 09 17:39:17.082436 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.082401    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 09 17:39:17.091107 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.091089    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:17.172189 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.172129    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal"
Jul 09 17:39:17.183030 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.183012    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 17:39:17.184664 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.184650    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 09 17:39:17.196706 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.196689    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 17:39:17.220858 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.220842    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 17:39:17.220962 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.220944    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 17:39:17.220999 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.220982    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 17:39:17.239350 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.239332    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:17.254516 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.254498    2573 apiserver.go:52] "Watching apiserver"
Jul 09 17:39:17.265728 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.265699    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 17:39:17.267026 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.266533    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-glm8m","openshift-multus/network-metrics-daemon-6qvsq","openshift-network-diagnostics/network-check-target-wtdtc","openshift-network-operator/iptables-alerter-7gvh8","openshift-ovn-kubernetes/ovnkube-node-64kgs","kube-system/global-pull-secret-syncer-mcsfz","kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rnsm2","openshift-dns/node-resolver-cctw4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal","openshift-multus/multus-ds27w","kube-system/konnectivity-agent-ps4sv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5","openshift-image-registry/node-ca-jbqs8"]
Jul 09 17:39:17.269283 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.269257    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.271440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.271419    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:17.271529 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.271492    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qvsq" podUID="8a62897b-ea47-4b2f-8762-a4fc7b7a18dc"
Jul 09 17:39:17.272021 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.271998    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 17:39:17.272093 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.272067    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 17:39:17.272206 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.272187    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 17:39:17.272318 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.272281    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zmhvw\""
Jul 09 17:39:17.272545 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.272530    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 17:39:17.273626 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.273610    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:17.273745 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.273668    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtdtc" podUID="fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae"
Jul 09 17:39:17.275756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.275740    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7gvh8"
Jul 09 17:39:17.276491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.276472    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-sys-fs\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.276554 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.276505    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk279\" (UniqueName: \"kubernetes.io/projected/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-kube-api-access-kk279\") pod \"network-metrics-daemon-6qvsq\" (UID: \"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc\") " pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:17.276554 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.276533    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pmjv\" (UniqueName: \"kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv\") pod \"network-check-target-wtdtc\" (UID: \"fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae\") " pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:17.276616 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.276558    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.276616 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.276579    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-socket-dir\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.276616 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.276602    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-registration-dir\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.276704 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.276635    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-device-dir\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.276704 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.276659    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-etc-selinux\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.276704 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.276680    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5qv\" (UniqueName: \"kubernetes.io/projected/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-kube-api-access-4r5qv\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.276830 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.276713    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs\") pod \"network-metrics-daemon-6qvsq\" (UID: \"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc\") " pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:17.277959 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.277942    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.278529 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.278505    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 17:39:17.278529 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.278517    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:39:17.278529 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.278509    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 17:39:17.278796 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.278783    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tgcnp\""
Jul 09 17:39:17.280187 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.280170    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:17.280265 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.280229    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcsfz" podUID="b282b3a8-fc56-4bfc-8626-947aada8ecde"
Jul 09 17:39:17.280607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.280592    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 17:39:17.280783 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.280769    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 17:39:17.280857 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.280836    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 17:39:17.280925 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.280867    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 17:39:17.280925 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.280873    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 17:39:17.282477 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.282458    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4w56t\""
Jul 09 17:39:17.282572 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.282458    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 17:39:17.282572 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.282462    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.282572 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.282565    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 17:39:17.284831 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.284807    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.285066 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.285049    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 17:39:17.285224 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.285208    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cfhd6\""
Jul 09 17:39:17.285395 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.285382    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 17:39:17.285395 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.285385    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 17:39:17.286565 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.286551    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 17:39:17.286638 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.286553    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 17:39:17.287050 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.287038    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cctw4"
Jul 09 17:39:17.287371 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.287358    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:39:17.287513 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.287502    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pgblv\""
Jul 09 17:39:17.287559 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.287528    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 17:39:17.289297 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.289283    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jbqs8"
Jul 09 17:39:17.289355 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.289342    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r2bwc\""
Jul 09 17:39:17.289707 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.289694    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 17:39:17.289880 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.289867    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 17:39:17.292443 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.292298    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.292443 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.292403    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 17:39:17.292582 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.292532    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 17:39:17.292977 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.292959    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 17:39:17.293043 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.293010    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rbh52\""
Jul 09 17:39:17.294599 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.294583    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ps4sv"
Jul 09 17:39:17.295324 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.295307    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 17:39:17.295573 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.295558    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nqnrf\""
Jul 09 17:39:17.297461 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.297445    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nbg5l\""
Jul 09 17:39:17.297590 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.297547    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 17:39:17.297669 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.297635    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 17:39:17.303004 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.302989    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tzndd"
Jul 09 17:39:17.310909 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.310895    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tzndd"
Jul 09 17:39:17.373469 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.373449    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 17:39:17.377085 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377069    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/46f1c9d1-6578-4343-88ba-900a5cc5bd28-multus-daemon-config\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.377182 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377090    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/26c0b0ef-78e7-455e-b5cf-52376c022784-agent-certs\") pod \"konnectivity-agent-ps4sv\" (UID: \"26c0b0ef-78e7-455e-b5cf-52376c022784\") " pod="kube-system/konnectivity-agent-ps4sv"
Jul 09 17:39:17.377182 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377107    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87v4n\" (UniqueName: \"kubernetes.io/projected/fcc10c4d-ba3f-4dd7-90ca-02826f1366b9-kube-api-access-87v4n\") pod \"iptables-alerter-7gvh8\" (UID: \"fcc10c4d-ba3f-4dd7-90ca-02826f1366b9\") " pod="openshift-network-operator/iptables-alerter-7gvh8"
Jul 09 17:39:17.377182 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377122    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-run-netns\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.377182 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377155    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/76f015b9-8aaa-40ee-a73e-492fb91418d0-serviceca\") pod \"node-ca-jbqs8\" (UID: \"76f015b9-8aaa-40ee-a73e-492fb91418d0\") " pod="openshift-image-registry/node-ca-jbqs8"
Jul 09 17:39:17.377182 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377179    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-sys-fs\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.377440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377195    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-var-lib-cni-multus\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.377440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377228    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-kubernetes\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.377440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377257    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-sys-fs\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.377440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377259    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-sys\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.377440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377303    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-multus-cni-dir\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.377440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6fb10245-3407-451c-a7fc-b4a6948ffb01-os-release\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.377440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377348    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6fb10245-3407-451c-a7fc-b4a6948ffb01-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.377440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377365    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-run-netns\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.377440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377379    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-systemd-units\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.377440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377393    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-run-openvswitch\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.377440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377416    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-env-overrides\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.377440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377438    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-run-ovn\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.378015 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377453    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-node-log\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.378015 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377471    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk279\" (UniqueName: \"kubernetes.io/projected/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-kube-api-access-kk279\") pod \"network-metrics-daemon-6qvsq\" (UID: \"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc\") " pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:17.378015 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377486    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-cnibin\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.378015 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377506    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-multus-socket-dir-parent\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.378015 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377526    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-kubelet\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.378015 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377597    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-sysctl-conf\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.378015 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377619    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-os-release\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.378015 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377641    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-etc-kubernetes\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.378015 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377660    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/26c0b0ef-78e7-455e-b5cf-52376c022784-konnectivity-ca\") pod \"konnectivity-agent-ps4sv\" (UID: \"26c0b0ef-78e7-455e-b5cf-52376c022784\") " pod="kube-system/konnectivity-agent-ps4sv"
Jul 09 17:39:17.378015 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377673    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-slash\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.378015 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377707    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46f1c9d1-6578-4343-88ba-900a5cc5bd28-cni-binary-copy\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.378015 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377756    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-systemd\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.378015 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377782    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-host\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.378015 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377800    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2ds8\" (UniqueName: \"kubernetes.io/projected/76f015b9-8aaa-40ee-a73e-492fb91418d0-kube-api-access-g2ds8\") pod \"node-ca-jbqs8\" (UID: \"76f015b9-8aaa-40ee-a73e-492fb91418d0\") " pod="openshift-image-registry/node-ca-jbqs8"
Jul 09 17:39:17.378491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377826    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-device-dir\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.378491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377847    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m827\" (UniqueName: \"kubernetes.io/projected/46f1c9d1-6578-4343-88ba-900a5cc5bd28-kube-api-access-6m827\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.378491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377862    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-run-systemd\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.378491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-etc-openvswitch\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.378491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377896    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-device-dir\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.378491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-log-socket\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.378491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377924    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.378491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377969    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-ovn-node-metrics-cert\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.378491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.377989    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fb10245-3407-451c-a7fc-b4a6948ffb01-system-cni-dir\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.378491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378019    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-ovnkube-script-lib\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.378491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378042    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3dcc235-2a55-4e67-a42c-4d047739f8f0-hosts-file\") pod \"node-resolver-cctw4\" (UID: \"a3dcc235-2a55-4e67-a42c-4d047739f8f0\") " pod="openshift-dns/node-resolver-cctw4"
Jul 09 17:39:17.378491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.378491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378078    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-etc-selinux\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.379106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378103    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fb10245-3407-451c-a7fc-b4a6948ffb01-tuning-conf-dir\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.379106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378117    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.379106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378126    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-run-k8s-cni-cncf-io\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.379106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378145    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-var-lib-openvswitch\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.379106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378161    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-sysctl-d\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.379106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378182    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-run\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.379106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378196    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feb7242f-ff85-407e-a02d-edca7d9241d1-tmp\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.379106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378213    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b282b3a8-fc56-4bfc-8626-947aada8ecde-kubelet-config\") pod \"global-pull-secret-syncer-mcsfz\" (UID: \"b282b3a8-fc56-4bfc-8626-947aada8ecde\") " pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:17.379106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378227    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmf2n\" (UniqueName: \"kubernetes.io/projected/feb7242f-ff85-407e-a02d-edca7d9241d1-kube-api-access-nmf2n\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.379106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378249    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-system-cni-dir\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.379106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378247    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-etc-selinux\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.379106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378280    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5qv\" (UniqueName: \"kubernetes.io/projected/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-kube-api-access-4r5qv\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.379106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378298    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcc10c4d-ba3f-4dd7-90ca-02826f1366b9-host-slash\") pod \"iptables-alerter-7gvh8\" (UID: \"fcc10c4d-ba3f-4dd7-90ca-02826f1366b9\") " pod="openshift-network-operator/iptables-alerter-7gvh8"
Jul 09 17:39:17.379106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378317    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-lib-modules\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.379806 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378337    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-tuned\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.379806 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378366    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fb10245-3407-451c-a7fc-b4a6948ffb01-cnibin\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.379806 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378400    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fb10245-3407-451c-a7fc-b4a6948ffb01-cni-binary-copy\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.379806 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378431    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79zm4\" (UniqueName: \"kubernetes.io/projected/6fb10245-3407-451c-a7fc-b4a6948ffb01-kube-api-access-79zm4\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.379806 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378455    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-run-multus-certs\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.379806 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378478    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-ovnkube-config\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.379806 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378511    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-sysconfig\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.379806 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378541    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs\") pod \"network-metrics-daemon-6qvsq\" (UID: \"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc\") " pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:17.379806 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378567    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6fb10245-3407-451c-a7fc-b4a6948ffb01-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.379806 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378591    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-hostroot\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.379806 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378612    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-multus-conf-dir\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.379806 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.378619    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:17.379806 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378635    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x49xx\" (UniqueName: \"kubernetes.io/projected/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-kube-api-access-x49xx\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.380460 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.378694    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs podName:8a62897b-ea47-4b2f-8762-a4fc7b7a18dc nodeName:}" failed. No retries permitted until 2026-07-09 17:39:17.878659859 +0000 UTC m=+1.892188970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs") pod "network-metrics-daemon-6qvsq" (UID: "8a62897b-ea47-4b2f-8762-a4fc7b7a18dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:17.380460 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378746    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret\") pod \"global-pull-secret-syncer-mcsfz\" (UID: \"b282b3a8-fc56-4bfc-8626-947aada8ecde\") " pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:17.380460 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378768    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fcc10c4d-ba3f-4dd7-90ca-02826f1366b9-iptables-alerter-script\") pod \"iptables-alerter-7gvh8\" (UID: \"fcc10c4d-ba3f-4dd7-90ca-02826f1366b9\") " pod="openshift-network-operator/iptables-alerter-7gvh8"
Jul 09 17:39:17.380460 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378785    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-run-ovn-kubernetes\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.380460 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378822    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-cni-bin\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.380460 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378844    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-modprobe-d\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.380460 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378859    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3dcc235-2a55-4e67-a42c-4d047739f8f0-tmp-dir\") pod \"node-resolver-cctw4\" (UID: \"a3dcc235-2a55-4e67-a42c-4d047739f8f0\") " pod="openshift-dns/node-resolver-cctw4"
Jul 09 17:39:17.380460 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378877    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmjv\" (UniqueName: \"kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv\") pod \"network-check-target-wtdtc\" (UID: \"fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae\") " pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:17.380460 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378892    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-socket-dir\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.380460 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378909    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-registration-dir\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.380460 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378923    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-cni-netd\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.380460 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378938    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-var-lib-kubelet\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.380460 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378953    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76f015b9-8aaa-40ee-a73e-492fb91418d0-host\") pod \"node-ca-jbqs8\" (UID: \"76f015b9-8aaa-40ee-a73e-492fb91418d0\") " pod="openshift-image-registry/node-ca-jbqs8"
Jul 09 17:39:17.381064 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378972    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b282b3a8-fc56-4bfc-8626-947aada8ecde-dbus\") pod \"global-pull-secret-syncer-mcsfz\" (UID: \"b282b3a8-fc56-4bfc-8626-947aada8ecde\") " pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:17.381064 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.378997    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-registration-dir\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.381064 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.379040    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-socket-dir\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.381064 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.379047    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-var-lib-cni-bin\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.381064 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.379470    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-var-lib-kubelet\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.381064 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.379487    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b6lq\" (UniqueName: \"kubernetes.io/projected/a3dcc235-2a55-4e67-a42c-4d047739f8f0-kube-api-access-2b6lq\") pod \"node-resolver-cctw4\" (UID: \"a3dcc235-2a55-4e67-a42c-4d047739f8f0\") " pod="openshift-dns/node-resolver-cctw4"
Jul 09 17:39:17.388908 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.388655    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 17:39:17.389539 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.389519    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:17.389669 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.389544    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:17.389669 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.389557    2573 projected.go:196] Error preparing data for projected volume kube-api-access-9pmjv for pod openshift-network-diagnostics/network-check-target-wtdtc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:17.389669 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.389651    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv podName:fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae nodeName:}" failed. No retries permitted until 2026-07-09 17:39:17.889633195 +0000 UTC m=+1.903162301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9pmjv" (UniqueName: "kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv") pod "network-check-target-wtdtc" (UID: "fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:17.391637 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.391616    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk279\" (UniqueName: \"kubernetes.io/projected/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-kube-api-access-kk279\") pod \"network-metrics-daemon-6qvsq\" (UID: \"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc\") " pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:17.391637 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.391629    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5qv\" (UniqueName: \"kubernetes.io/projected/8203ebfb-2681-4fba-9f3d-9aa98f4ecfee-kube-api-access-4r5qv\") pod \"aws-ebs-csi-driver-node-7mfh5\" (UID: \"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.400995 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.400955    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:34:16 +0000 UTC" deadline="2028-02-03 19:26:43.871006409 +0000 UTC"
Jul 09 17:39:17.400995 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.400991    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13777h47m26.470017636s"
Jul 09 17:39:17.479709 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.479687    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-host\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.479826 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.479728    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ds8\" (UniqueName: \"kubernetes.io/projected/76f015b9-8aaa-40ee-a73e-492fb91418d0-kube-api-access-g2ds8\") pod \"node-ca-jbqs8\" (UID: \"76f015b9-8aaa-40ee-a73e-492fb91418d0\") " pod="openshift-image-registry/node-ca-jbqs8"
Jul 09 17:39:17.479826 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.479746    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m827\" (UniqueName: \"kubernetes.io/projected/46f1c9d1-6578-4343-88ba-900a5cc5bd28-kube-api-access-6m827\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.479826 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.479768    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-run-systemd\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.479826 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.479793    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-etc-openvswitch\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.479826 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.479809    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-log-socket\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.480024 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.479826    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.480024 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.479834    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-host\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.480024 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.479849    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-ovn-node-metrics-cert\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.480024 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.479881    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fb10245-3407-451c-a7fc-b4a6948ffb01-system-cni-dir\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.480024 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.479901    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-ovnkube-script-lib\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.480024 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.479981    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fb10245-3407-451c-a7fc-b4a6948ffb01-system-cni-dir\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.480024 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.479985    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-log-socket\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.480024 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480019    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-run-systemd\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.480371 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480041    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.480371 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480068    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3dcc235-2a55-4e67-a42c-4d047739f8f0-hosts-file\") pod \"node-resolver-cctw4\" (UID: \"a3dcc235-2a55-4e67-a42c-4d047739f8f0\") " pod="openshift-dns/node-resolver-cctw4"
Jul 09 17:39:17.480371 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480086    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-etc-openvswitch\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.480371 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480117    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fb10245-3407-451c-a7fc-b4a6948ffb01-tuning-conf-dir\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.480371 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480146    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3dcc235-2a55-4e67-a42c-4d047739f8f0-hosts-file\") pod \"node-resolver-cctw4\" (UID: \"a3dcc235-2a55-4e67-a42c-4d047739f8f0\") " pod="openshift-dns/node-resolver-cctw4"
Jul 09 17:39:17.480371 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480152    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-run-k8s-cni-cncf-io\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.480371 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480180    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-var-lib-openvswitch\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.480371 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480203    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-sysctl-d\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.480371 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480225    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-run\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.480371 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480229    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-run-k8s-cni-cncf-io\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.480371 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480260    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feb7242f-ff85-407e-a02d-edca7d9241d1-tmp\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.480371 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480284    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-run\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.480371 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480289    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fb10245-3407-451c-a7fc-b4a6948ffb01-tuning-conf-dir\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.480371 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480291    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b282b3a8-fc56-4bfc-8626-947aada8ecde-kubelet-config\") pod \"global-pull-secret-syncer-mcsfz\" (UID: \"b282b3a8-fc56-4bfc-8626-947aada8ecde\") " pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:17.481023 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480330    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmf2n\" (UniqueName: \"kubernetes.io/projected/feb7242f-ff85-407e-a02d-edca7d9241d1-kube-api-access-nmf2n\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.481023 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480331    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b282b3a8-fc56-4bfc-8626-947aada8ecde-kubelet-config\") pod \"global-pull-secret-syncer-mcsfz\" (UID: \"b282b3a8-fc56-4bfc-8626-947aada8ecde\") " pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:17.481023 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480334    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-sysctl-d\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.481023 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480354    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-system-cni-dir\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.481023 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480405    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-system-cni-dir\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.481023 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480507    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcc10c4d-ba3f-4dd7-90ca-02826f1366b9-host-slash\") pod \"iptables-alerter-7gvh8\" (UID: \"fcc10c4d-ba3f-4dd7-90ca-02826f1366b9\") " pod="openshift-network-operator/iptables-alerter-7gvh8"
Jul 09 17:39:17.481023 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480535    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-lib-modules\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.481023 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-tuned\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.481023 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480580    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcc10c4d-ba3f-4dd7-90ca-02826f1366b9-host-slash\") pod \"iptables-alerter-7gvh8\" (UID: \"fcc10c4d-ba3f-4dd7-90ca-02826f1366b9\") " pod="openshift-network-operator/iptables-alerter-7gvh8"
Jul 09 17:39:17.481023 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480555    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-ovnkube-script-lib\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.481023 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480583    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fb10245-3407-451c-a7fc-b4a6948ffb01-cnibin\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.481023 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480625    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fb10245-3407-451c-a7fc-b4a6948ffb01-cni-binary-copy\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.481023 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480647    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fb10245-3407-451c-a7fc-b4a6948ffb01-cnibin\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.481023 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480652    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-lib-modules\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.481756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480652    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79zm4\" (UniqueName: \"kubernetes.io/projected/6fb10245-3407-451c-a7fc-b4a6948ffb01-kube-api-access-79zm4\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.481756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480711    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-run-multus-certs\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.481756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480755    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-ovnkube-config\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.481756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480780    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-sysconfig\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.481756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480818    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6fb10245-3407-451c-a7fc-b4a6948ffb01-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.481756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480798    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-var-lib-openvswitch\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.481756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480844    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-hostroot\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.481756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480894    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-sysconfig\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.481756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480925    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-multus-conf-dir\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.481756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480961    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x49xx\" (UniqueName: \"kubernetes.io/projected/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-kube-api-access-x49xx\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.481756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480991    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-run-multus-certs\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.481756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.480964    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-multus-conf-dir\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.481756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481033    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret\") pod \"global-pull-secret-syncer-mcsfz\" (UID: \"b282b3a8-fc56-4bfc-8626-947aada8ecde\") " pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:17.481756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481065    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fcc10c4d-ba3f-4dd7-90ca-02826f1366b9-iptables-alerter-script\") pod \"iptables-alerter-7gvh8\" (UID: \"fcc10c4d-ba3f-4dd7-90ca-02826f1366b9\") " pod="openshift-network-operator/iptables-alerter-7gvh8"
Jul 09 17:39:17.482445 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-run-ovn-kubernetes\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.482445 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481122    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-cni-bin\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.482445 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481144    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-modprobe-d\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.482445 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.481152    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:17.482445 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481166    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3dcc235-2a55-4e67-a42c-4d047739f8f0-tmp-dir\") pod \"node-resolver-cctw4\" (UID: \"a3dcc235-2a55-4e67-a42c-4d047739f8f0\") " pod="openshift-dns/node-resolver-cctw4"
Jul 09 17:39:17.482445 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481183    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fb10245-3407-451c-a7fc-b4a6948ffb01-cni-binary-copy\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.482445 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481205    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-cni-netd\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.482445 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.481218    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret podName:b282b3a8-fc56-4bfc-8626-947aada8ecde nodeName:}" failed. No retries permitted until 2026-07-09 17:39:17.981201907 +0000 UTC m=+1.994731017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret") pod "global-pull-secret-syncer-mcsfz" (UID: "b282b3a8-fc56-4bfc-8626-947aada8ecde") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:17.482445 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481253    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-cni-netd\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.482445 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481261    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-var-lib-kubelet\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.482445 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481273    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-run-ovn-kubernetes\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.482445 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481286    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-ovnkube-config\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.482445 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481289    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76f015b9-8aaa-40ee-a73e-492fb91418d0-host\") pod \"node-ca-jbqs8\" (UID: \"76f015b9-8aaa-40ee-a73e-492fb91418d0\") " pod="openshift-image-registry/node-ca-jbqs8"
Jul 09 17:39:17.482445 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481326    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-cni-bin\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.483167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481330    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b282b3a8-fc56-4bfc-8626-947aada8ecde-dbus\") pod \"global-pull-secret-syncer-mcsfz\" (UID: \"b282b3a8-fc56-4bfc-8626-947aada8ecde\") " pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:17.483167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481373    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-modprobe-d\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.483167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481388    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b282b3a8-fc56-4bfc-8626-947aada8ecde-dbus\") pod \"global-pull-secret-syncer-mcsfz\" (UID: \"b282b3a8-fc56-4bfc-8626-947aada8ecde\") " pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:17.483167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481393    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76f015b9-8aaa-40ee-a73e-492fb91418d0-host\") pod \"node-ca-jbqs8\" (UID: \"76f015b9-8aaa-40ee-a73e-492fb91418d0\") " pod="openshift-image-registry/node-ca-jbqs8"
Jul 09 17:39:17.483167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481417    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-var-lib-cni-bin\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.483167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-var-lib-kubelet\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.483167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481453    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-var-lib-cni-bin\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.483167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481420    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-var-lib-kubelet\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.483167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481467    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b6lq\" (UniqueName: \"kubernetes.io/projected/a3dcc235-2a55-4e67-a42c-4d047739f8f0-kube-api-access-2b6lq\") pod \"node-resolver-cctw4\" (UID: \"a3dcc235-2a55-4e67-a42c-4d047739f8f0\") " pod="openshift-dns/node-resolver-cctw4"
Jul 09 17:39:17.483167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481509    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-var-lib-kubelet\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.483167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481511    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/46f1c9d1-6578-4343-88ba-900a5cc5bd28-multus-daemon-config\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.483167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481546    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/26c0b0ef-78e7-455e-b5cf-52376c022784-agent-certs\") pod \"konnectivity-agent-ps4sv\" (UID: \"26c0b0ef-78e7-455e-b5cf-52376c022784\") " pod="kube-system/konnectivity-agent-ps4sv"
Jul 09 17:39:17.483167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481574    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87v4n\" (UniqueName: \"kubernetes.io/projected/fcc10c4d-ba3f-4dd7-90ca-02826f1366b9-kube-api-access-87v4n\") pod \"iptables-alerter-7gvh8\" (UID: \"fcc10c4d-ba3f-4dd7-90ca-02826f1366b9\") " pod="openshift-network-operator/iptables-alerter-7gvh8"
Jul 09 17:39:17.483167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481595    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3dcc235-2a55-4e67-a42c-4d047739f8f0-tmp-dir\") pod \"node-resolver-cctw4\" (UID: \"a3dcc235-2a55-4e67-a42c-4d047739f8f0\") " pod="openshift-dns/node-resolver-cctw4"
Jul 09 17:39:17.483167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481629    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-run-netns\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.483868 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-run-netns\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.483868 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481702    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/76f015b9-8aaa-40ee-a73e-492fb91418d0-serviceca\") pod \"node-ca-jbqs8\" (UID: \"76f015b9-8aaa-40ee-a73e-492fb91418d0\") " pod="openshift-image-registry/node-ca-jbqs8"
Jul 09 17:39:17.483868 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481753    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-var-lib-cni-multus\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.483868 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481784    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-kubernetes\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.483868 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481809    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-sys\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.483868 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481834    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-multus-cni-dir\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.483868 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481862    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6fb10245-3407-451c-a7fc-b4a6948ffb01-os-release\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.483868 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481897    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6fb10245-3407-451c-a7fc-b4a6948ffb01-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.483868 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481925    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-run-netns\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.483868 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481943    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-kubernetes\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.483868 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481951    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-systemd-units\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.483868 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.481987    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-run-openvswitch\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.483868 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482003    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-var-lib-cni-multus\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.483868 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482019    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/46f1c9d1-6578-4343-88ba-900a5cc5bd28-multus-daemon-config\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.483868 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482029    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-env-overrides\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.484319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482057    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-run-ovn\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.484319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482079    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-systemd-units\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.484319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482085    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-node-log\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.484319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482121    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-host-run-netns\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.484319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482118    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-cnibin\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.484319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482183    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-multus-socket-dir-parent\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.484319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482186    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-multus-cni-dir\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.484319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482189    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-hostroot\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.484319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482205    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-kubelet\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.484319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482232    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-sysctl-conf\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.484319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482235    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-sys\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.484319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482268    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6fb10245-3407-451c-a7fc-b4a6948ffb01-os-release\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.484319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482276    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-os-release\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.484319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482281    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-run-ovn\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.484319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482302    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-etc-kubernetes\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.484730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482328    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-multus-socket-dir-parent\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.484730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482332    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-cnibin\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.484730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482331    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/26c0b0ef-78e7-455e-b5cf-52376c022784-konnectivity-ca\") pod \"konnectivity-agent-ps4sv\" (UID: \"26c0b0ef-78e7-455e-b5cf-52376c022784\") " pod="kube-system/konnectivity-agent-ps4sv"
Jul 09 17:39:17.484730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482371    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-slash\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.484730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482384    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-kubelet\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.484730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482415    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46f1c9d1-6578-4343-88ba-900a5cc5bd28-cni-binary-copy\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.484730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-systemd\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.484730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482506    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-systemd\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.484730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482505    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-sysctl-conf\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.484730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482542    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-host-slash\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.484730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482795    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-node-log\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.484730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482886    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/76f015b9-8aaa-40ee-a73e-492fb91418d0-serviceca\") pod \"node-ca-jbqs8\" (UID: \"76f015b9-8aaa-40ee-a73e-492fb91418d0\") " pod="openshift-image-registry/node-ca-jbqs8"
Jul 09 17:39:17.484730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482936    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-env-overrides\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.484730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482950    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-os-release\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.484730 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.482985    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f1c9d1-6578-4343-88ba-900a5cc5bd28-etc-kubernetes\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.485175 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.483020    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-run-openvswitch\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.485175 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.483030    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feb7242f-ff85-407e-a02d-edca7d9241d1-tmp\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.485175 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.483132    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6fb10245-3407-451c-a7fc-b4a6948ffb01-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.485175 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.483146    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fcc10c4d-ba3f-4dd7-90ca-02826f1366b9-iptables-alerter-script\") pod \"iptables-alerter-7gvh8\" (UID: \"fcc10c4d-ba3f-4dd7-90ca-02826f1366b9\") " pod="openshift-network-operator/iptables-alerter-7gvh8"
Jul 09 17:39:17.485175 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.483294    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-ovn-node-metrics-cert\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.485175 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.483439    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46f1c9d1-6578-4343-88ba-900a5cc5bd28-cni-binary-copy\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.485175 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.483479    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/feb7242f-ff85-407e-a02d-edca7d9241d1-etc-tuned\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.485175 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.483946    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6fb10245-3407-451c-a7fc-b4a6948ffb01-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.485175 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.484023    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/26c0b0ef-78e7-455e-b5cf-52376c022784-agent-certs\") pod \"konnectivity-agent-ps4sv\" (UID: \"26c0b0ef-78e7-455e-b5cf-52376c022784\") " pod="kube-system/konnectivity-agent-ps4sv"
Jul 09 17:39:17.485175 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.484091    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/26c0b0ef-78e7-455e-b5cf-52376c022784-konnectivity-ca\") pod \"konnectivity-agent-ps4sv\" (UID: \"26c0b0ef-78e7-455e-b5cf-52376c022784\") " pod="kube-system/konnectivity-agent-ps4sv"
Jul 09 17:39:17.491179 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.491150    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x49xx\" (UniqueName: \"kubernetes.io/projected/fd0f26b5-5eac-453c-ba70-2cdef4bd156d-kube-api-access-x49xx\") pod \"ovnkube-node-64kgs\" (UID: \"fd0f26b5-5eac-453c-ba70-2cdef4bd156d\") " pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.491179 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.491164    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m827\" (UniqueName: \"kubernetes.io/projected/46f1c9d1-6578-4343-88ba-900a5cc5bd28-kube-api-access-6m827\") pod \"multus-ds27w\" (UID: \"46f1c9d1-6578-4343-88ba-900a5cc5bd28\") " pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.491364 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.491185    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmf2n\" (UniqueName: \"kubernetes.io/projected/feb7242f-ff85-407e-a02d-edca7d9241d1-kube-api-access-nmf2n\") pod \"tuned-rnsm2\" (UID: \"feb7242f-ff85-407e-a02d-edca7d9241d1\") " pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.491549 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.491531    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79zm4\" (UniqueName: \"kubernetes.io/projected/6fb10245-3407-451c-a7fc-b4a6948ffb01-kube-api-access-79zm4\") pod \"multus-additional-cni-plugins-glm8m\" (UID: \"6fb10245-3407-451c-a7fc-b4a6948ffb01\") " pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.492511 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.492490    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87v4n\" (UniqueName: \"kubernetes.io/projected/fcc10c4d-ba3f-4dd7-90ca-02826f1366b9-kube-api-access-87v4n\") pod \"iptables-alerter-7gvh8\" (UID: \"fcc10c4d-ba3f-4dd7-90ca-02826f1366b9\") " pod="openshift-network-operator/iptables-alerter-7gvh8"
Jul 09 17:39:17.492999 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.492981    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2ds8\" (UniqueName: \"kubernetes.io/projected/76f015b9-8aaa-40ee-a73e-492fb91418d0-kube-api-access-g2ds8\") pod \"node-ca-jbqs8\" (UID: \"76f015b9-8aaa-40ee-a73e-492fb91418d0\") " pod="openshift-image-registry/node-ca-jbqs8"
Jul 09 17:39:17.493317 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.493302    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b6lq\" (UniqueName: \"kubernetes.io/projected/a3dcc235-2a55-4e67-a42c-4d047739f8f0-kube-api-access-2b6lq\") pod \"node-resolver-cctw4\" (UID: \"a3dcc235-2a55-4e67-a42c-4d047739f8f0\") " pod="openshift-dns/node-resolver-cctw4"
Jul 09 17:39:17.533918 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.533890    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:17.595677 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.595651    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5"
Jul 09 17:39:17.601713 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:17.601682    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343c2c99cf08d1b78e530233805856b4.slice/crio-ca0ceb1a9f9d61309dd393a05db0c61fec0e627b58afdae0b6d1a0b4ea7f5ff1 WatchSource:0}: Error finding container ca0ceb1a9f9d61309dd393a05db0c61fec0e627b58afdae0b6d1a0b4ea7f5ff1: Status 404 returned error can't find the container with id ca0ceb1a9f9d61309dd393a05db0c61fec0e627b58afdae0b6d1a0b4ea7f5ff1
Jul 09 17:39:17.602790 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:17.602765    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3b8930aa1846297e408b672c61a6ab.slice/crio-25632bd18f05efdc6daba4b8aaab2c124dec03569e6d7dcd5cb03ee9befdf67f WatchSource:0}: Error finding container 25632bd18f05efdc6daba4b8aaab2c124dec03569e6d7dcd5cb03ee9befdf67f: Status 404 returned error can't find the container with id 25632bd18f05efdc6daba4b8aaab2c124dec03569e6d7dcd5cb03ee9befdf67f
Jul 09 17:39:17.607634 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.607620    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:39:17.607782 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:17.607764    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8203ebfb_2681_4fba_9f3d_9aa98f4ecfee.slice/crio-ec0bfe69a7abc705058af008bff028a1aa2f2cf1401fc980d1d85ba0bfdd3597 WatchSource:0}: Error finding container ec0bfe69a7abc705058af008bff028a1aa2f2cf1401fc980d1d85ba0bfdd3597: Status 404 returned error can't find the container with id ec0bfe69a7abc705058af008bff028a1aa2f2cf1401fc980d1d85ba0bfdd3597
Jul 09 17:39:17.614647 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.614628    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7gvh8"
Jul 09 17:39:17.622175 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.622156    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ps4sv"
Jul 09 17:39:17.623986 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:17.623964    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc10c4d_ba3f_4dd7_90ca_02826f1366b9.slice/crio-112daf4523d86556755be00639c16784fe8b056a1d48bf17a11de0debb84561f WatchSource:0}: Error finding container 112daf4523d86556755be00639c16784fe8b056a1d48bf17a11de0debb84561f: Status 404 returned error can't find the container with id 112daf4523d86556755be00639c16784fe8b056a1d48bf17a11de0debb84561f
Jul 09 17:39:17.631243 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:17.631224    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c0b0ef_78e7_455e_b5cf_52376c022784.slice/crio-59e33d8cd2535268efcf989b92694ec97175db0759847652e71b062f122b287b WatchSource:0}: Error finding container 59e33d8cd2535268efcf989b92694ec97175db0759847652e71b062f122b287b: Status 404 returned error can't find the container with id 59e33d8cd2535268efcf989b92694ec97175db0759847652e71b062f122b287b
Jul 09 17:39:17.634972 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.634957    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:17.642123 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:17.642106    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0f26b5_5eac_453c_ba70_2cdef4bd156d.slice/crio-445090471f086269d835acff5a19cfbe0b158738a7dc56562a96cccb06a15d8d WatchSource:0}: Error finding container 445090471f086269d835acff5a19cfbe0b158738a7dc56562a96cccb06a15d8d: Status 404 returned error can't find the container with id 445090471f086269d835acff5a19cfbe0b158738a7dc56562a96cccb06a15d8d
Jul 09 17:39:17.648948 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.648933    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-glm8m"
Jul 09 17:39:17.656034 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:17.656017    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb10245_3407_451c_a7fc_b4a6948ffb01.slice/crio-04e47d04529b958f95034078e0bf583bbf179f60995b7d3c0949bc24c8f1c767 WatchSource:0}: Error finding container 04e47d04529b958f95034078e0bf583bbf179f60995b7d3c0949bc24c8f1c767: Status 404 returned error can't find the container with id 04e47d04529b958f95034078e0bf583bbf179f60995b7d3c0949bc24c8f1c767
Jul 09 17:39:17.663813 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.663796    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rnsm2"
Jul 09 17:39:17.665607 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.665594    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cctw4"
Jul 09 17:39:17.671487 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:17.671460    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb7242f_ff85_407e_a02d_edca7d9241d1.slice/crio-5cc76752c7e874cbea1cbd4df2bbcc285685710eb6b5d85219f13f7d9f4b3924 WatchSource:0}: Error finding container 5cc76752c7e874cbea1cbd4df2bbcc285685710eb6b5d85219f13f7d9f4b3924: Status 404 returned error can't find the container with id 5cc76752c7e874cbea1cbd4df2bbcc285685710eb6b5d85219f13f7d9f4b3924
Jul 09 17:39:17.674160 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:17.674143    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3dcc235_2a55_4e67_a42c_4d047739f8f0.slice/crio-494bdadd6c91556800536bd153a82bcf99e313532795dd5c3c43fe7ec3e43541 WatchSource:0}: Error finding container 494bdadd6c91556800536bd153a82bcf99e313532795dd5c3c43fe7ec3e43541: Status 404 returned error can't find the container with id 494bdadd6c91556800536bd153a82bcf99e313532795dd5c3c43fe7ec3e43541
Jul 09 17:39:17.682909 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.682894    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jbqs8"
Jul 09 17:39:17.686336 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.686318    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ds27w"
Jul 09 17:39:17.693040 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:17.693016    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f015b9_8aaa_40ee_a73e_492fb91418d0.slice/crio-4d98c341ff961a75de9c3697467ed09b595dcd0c9ee21f4ccdf112d42fc8a267 WatchSource:0}: Error finding container 4d98c341ff961a75de9c3697467ed09b595dcd0c9ee21f4ccdf112d42fc8a267: Status 404 returned error can't find the container with id 4d98c341ff961a75de9c3697467ed09b595dcd0c9ee21f4ccdf112d42fc8a267
Jul 09 17:39:17.695450 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:17.695435    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f1c9d1_6578_4343_88ba_900a5cc5bd28.slice/crio-124552f948e6b7c45753ea1c0944b2cefccb469566e4bdf0e63ab0a00a530b3a WatchSource:0}: Error finding container 124552f948e6b7c45753ea1c0944b2cefccb469566e4bdf0e63ab0a00a530b3a: Status 404 returned error can't find the container with id 124552f948e6b7c45753ea1c0944b2cefccb469566e4bdf0e63ab0a00a530b3a
Jul 09 17:39:17.696950 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.696933    2573 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-multus start failed in pod multus-ds27w_openshift-multus(46f1c9d1-6578-4343-88ba-900a5cc5bd28): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 09 17:39:17.696994 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.696966    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-ds27w" podUID="46f1c9d1-6578-4343-88ba-900a5cc5bd28"
Jul 09 17:39:17.885732 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.885693    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs\") pod \"network-metrics-daemon-6qvsq\" (UID: \"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc\") " pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:17.885908 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.885867    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:17.885972 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.885945    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs podName:8a62897b-ea47-4b2f-8762-a4fc7b7a18dc nodeName:}" failed. No retries permitted until 2026-07-09 17:39:18.885923794 +0000 UTC m=+2.899452902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs") pod "network-metrics-daemon-6qvsq" (UID: "8a62897b-ea47-4b2f-8762-a4fc7b7a18dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:17.986494 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.986458    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret\") pod \"global-pull-secret-syncer-mcsfz\" (UID: \"b282b3a8-fc56-4bfc-8626-947aada8ecde\") " pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:17.986494 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:17.986498    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmjv\" (UniqueName: \"kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv\") pod \"network-check-target-wtdtc\" (UID: \"fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae\") " pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:17.986812 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.986604    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:17.986812 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.986625    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:17.986812 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.986644    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:17.986812 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.986658    2573 projected.go:196] Error preparing data for projected volume kube-api-access-9pmjv for pod openshift-network-diagnostics/network-check-target-wtdtc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:17.986812 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.986663    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret podName:b282b3a8-fc56-4bfc-8626-947aada8ecde nodeName:}" failed. No retries permitted until 2026-07-09 17:39:18.98664928 +0000 UTC m=+3.000178375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret") pod "global-pull-secret-syncer-mcsfz" (UID: "b282b3a8-fc56-4bfc-8626-947aada8ecde") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:17.986812 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:17.986713    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv podName:fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae nodeName:}" failed. No retries permitted until 2026-07-09 17:39:18.986688974 +0000 UTC m=+3.000218093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pmjv" (UniqueName: "kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv") pod "network-check-target-wtdtc" (UID: "fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:18.314767 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.314138    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:34:17 +0000 UTC" deadline="2028-01-18 01:18:15.563804063 +0000 UTC"
Jul 09 17:39:18.314767 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.314197    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13375h38m57.249611798s"
Jul 09 17:39:18.478239 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.478145    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:18.524979 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.524922    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:18.525133 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:18.525061    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qvsq" podUID="8a62897b-ea47-4b2f-8762-a4fc7b7a18dc"
Jul 09 17:39:18.525205 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.525138    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:18.525252 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:18.525206    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtdtc" podUID="fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae"
Jul 09 17:39:18.539577 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.539526    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ds27w" event={"ID":"46f1c9d1-6578-4343-88ba-900a5cc5bd28","Type":"ContainerStarted","Data":"124552f948e6b7c45753ea1c0944b2cefccb469566e4bdf0e63ab0a00a530b3a"}
Jul 09 17:39:18.548120 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:18.548088    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5228585713df6fe73e63d85efc7cb37f2a929792a1afb637a77bf56a4f2a40f9\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-ds27w" podUID="46f1c9d1-6578-4343-88ba-900a5cc5bd28"
Jul 09 17:39:18.564482 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.564411    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jbqs8" event={"ID":"76f015b9-8aaa-40ee-a73e-492fb91418d0","Type":"ContainerStarted","Data":"4d98c341ff961a75de9c3697467ed09b595dcd0c9ee21f4ccdf112d42fc8a267"}
Jul 09 17:39:18.578205 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.578128    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs" event={"ID":"fd0f26b5-5eac-453c-ba70-2cdef4bd156d","Type":"ContainerStarted","Data":"445090471f086269d835acff5a19cfbe0b158738a7dc56562a96cccb06a15d8d"}
Jul 09 17:39:18.590292 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.590255    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ps4sv" event={"ID":"26c0b0ef-78e7-455e-b5cf-52376c022784","Type":"ContainerStarted","Data":"59e33d8cd2535268efcf989b92694ec97175db0759847652e71b062f122b287b"}
Jul 09 17:39:18.594578 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.594544    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5" event={"ID":"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee","Type":"ContainerStarted","Data":"ec0bfe69a7abc705058af008bff028a1aa2f2cf1401fc980d1d85ba0bfdd3597"}
Jul 09 17:39:18.602905 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.602874    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal" event={"ID":"0b3b8930aa1846297e408b672c61a6ab","Type":"ContainerStarted","Data":"25632bd18f05efdc6daba4b8aaab2c124dec03569e6d7dcd5cb03ee9befdf67f"}
Jul 09 17:39:18.617922 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.617845    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal" event={"ID":"343c2c99cf08d1b78e530233805856b4","Type":"ContainerStarted","Data":"ca0ceb1a9f9d61309dd393a05db0c61fec0e627b58afdae0b6d1a0b4ea7f5ff1"}
Jul 09 17:39:18.621839 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.621810    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cctw4" event={"ID":"a3dcc235-2a55-4e67-a42c-4d047739f8f0","Type":"ContainerStarted","Data":"494bdadd6c91556800536bd153a82bcf99e313532795dd5c3c43fe7ec3e43541"}
Jul 09 17:39:18.627500 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.627473    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rnsm2" event={"ID":"feb7242f-ff85-407e-a02d-edca7d9241d1","Type":"ContainerStarted","Data":"5cc76752c7e874cbea1cbd4df2bbcc285685710eb6b5d85219f13f7d9f4b3924"}
Jul 09 17:39:18.640336 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.640014    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glm8m" event={"ID":"6fb10245-3407-451c-a7fc-b4a6948ffb01","Type":"ContainerStarted","Data":"04e47d04529b958f95034078e0bf583bbf179f60995b7d3c0949bc24c8f1c767"}
Jul 09 17:39:18.655639 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.655604    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7gvh8" event={"ID":"fcc10c4d-ba3f-4dd7-90ca-02826f1366b9","Type":"ContainerStarted","Data":"112daf4523d86556755be00639c16784fe8b056a1d48bf17a11de0debb84561f"}
Jul 09 17:39:18.893328 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.893243    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs\") pod \"network-metrics-daemon-6qvsq\" (UID: \"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc\") " pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:18.893492 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:18.893436    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:18.893548 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:18.893498    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs podName:8a62897b-ea47-4b2f-8762-a4fc7b7a18dc nodeName:}" failed. No retries permitted until 2026-07-09 17:39:20.893479313 +0000 UTC m=+4.907008426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs") pod "network-metrics-daemon-6qvsq" (UID: "8a62897b-ea47-4b2f-8762-a4fc7b7a18dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:18.993833 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.993794    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret\") pod \"global-pull-secret-syncer-mcsfz\" (UID: \"b282b3a8-fc56-4bfc-8626-947aada8ecde\") " pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:18.994011 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:18.993846    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmjv\" (UniqueName: \"kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv\") pod \"network-check-target-wtdtc\" (UID: \"fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae\") " pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:18.994011 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:18.993984    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:18.994011 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:18.994001    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:18.994011 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:18.994013    2573 projected.go:196] Error preparing data for projected volume kube-api-access-9pmjv for pod openshift-network-diagnostics/network-check-target-wtdtc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:18.994213 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:18.994069    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv podName:fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae nodeName:}" failed. No retries permitted until 2026-07-09 17:39:20.994051758 +0000 UTC m=+5.007580868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pmjv" (UniqueName: "kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv") pod "network-check-target-wtdtc" (UID: "fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:18.994213 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:18.994085    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:18.994213 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:18.994166    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret podName:b282b3a8-fc56-4bfc-8626-947aada8ecde nodeName:}" failed. No retries permitted until 2026-07-09 17:39:20.9941307 +0000 UTC m=+5.007659797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret") pod "global-pull-secret-syncer-mcsfz" (UID: "b282b3a8-fc56-4bfc-8626-947aada8ecde") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:19.315809 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:19.315197    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:34:17 +0000 UTC" deadline="2028-02-05 17:11:04.702573229 +0000 UTC"
Jul 09 17:39:19.315809 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:19.315253    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13823h31m45.38732436s"
Jul 09 17:39:19.423008 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:19.422937    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:19.522821 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:19.522785    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:19.522997 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:19.522926    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcsfz" podUID="b282b3a8-fc56-4bfc-8626-947aada8ecde"
Jul 09 17:39:20.525771 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:20.525044    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:20.525771 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:20.525414    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qvsq" podUID="8a62897b-ea47-4b2f-8762-a4fc7b7a18dc"
Jul 09 17:39:20.525771 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:20.525693    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:20.526321 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:20.525823    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtdtc" podUID="fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae"
Jul 09 17:39:20.918708 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:20.918619    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs\") pod \"network-metrics-daemon-6qvsq\" (UID: \"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc\") " pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:20.918880 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:20.918786    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:20.918880 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:20.918858    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs podName:8a62897b-ea47-4b2f-8762-a4fc7b7a18dc nodeName:}" failed. No retries permitted until 2026-07-09 17:39:24.918840488 +0000 UTC m=+8.932369583 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs") pod "network-metrics-daemon-6qvsq" (UID: "8a62897b-ea47-4b2f-8762-a4fc7b7a18dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:21.019953 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:21.019914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret\") pod \"global-pull-secret-syncer-mcsfz\" (UID: \"b282b3a8-fc56-4bfc-8626-947aada8ecde\") " pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:21.020127 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:21.019968    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmjv\" (UniqueName: \"kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv\") pod \"network-check-target-wtdtc\" (UID: \"fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae\") " pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:21.020190 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:21.020133    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:21.020190 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:21.020151    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:21.020190 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:21.020164    2573 projected.go:196] Error preparing data for projected volume kube-api-access-9pmjv for pod openshift-network-diagnostics/network-check-target-wtdtc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:21.020352 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:21.020217    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv podName:fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae nodeName:}" failed. No retries permitted until 2026-07-09 17:39:25.020198031 +0000 UTC m=+9.033727124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pmjv" (UniqueName: "kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv") pod "network-check-target-wtdtc" (UID: "fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:21.020604 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:21.020512    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:21.020604 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:21.020579    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret podName:b282b3a8-fc56-4bfc-8626-947aada8ecde nodeName:}" failed. No retries permitted until 2026-07-09 17:39:25.020561826 +0000 UTC m=+9.034090939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret") pod "global-pull-secret-syncer-mcsfz" (UID: "b282b3a8-fc56-4bfc-8626-947aada8ecde") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:21.522834 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:21.522794    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:21.523003 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:21.522939    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcsfz" podUID="b282b3a8-fc56-4bfc-8626-947aada8ecde"
Jul 09 17:39:22.522773 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:22.522743    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:22.523246 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:22.522782    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:22.523246 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:22.522869    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtdtc" podUID="fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae"
Jul 09 17:39:22.523388 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:22.523272    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qvsq" podUID="8a62897b-ea47-4b2f-8762-a4fc7b7a18dc"
Jul 09 17:39:23.522403 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:23.522370    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:23.522552 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:23.522492    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcsfz" podUID="b282b3a8-fc56-4bfc-8626-947aada8ecde"
Jul 09 17:39:24.523228 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:24.523165    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:24.523228 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:24.523165    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:24.523781 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:24.523325    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qvsq" podUID="8a62897b-ea47-4b2f-8762-a4fc7b7a18dc"
Jul 09 17:39:24.523781 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:24.523595    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtdtc" podUID="fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae"
Jul 09 17:39:24.946427 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:24.946347    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs\") pod \"network-metrics-daemon-6qvsq\" (UID: \"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc\") " pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:24.946573 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:24.946498    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:24.946573 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:24.946567    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs podName:8a62897b-ea47-4b2f-8762-a4fc7b7a18dc nodeName:}" failed. No retries permitted until 2026-07-09 17:39:32.946548537 +0000 UTC m=+16.960077642 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs") pod "network-metrics-daemon-6qvsq" (UID: "8a62897b-ea47-4b2f-8762-a4fc7b7a18dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:25.047764 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:25.047709    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmjv\" (UniqueName: \"kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv\") pod \"network-check-target-wtdtc\" (UID: \"fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae\") " pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:25.047935 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:25.047814    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret\") pod \"global-pull-secret-syncer-mcsfz\" (UID: \"b282b3a8-fc56-4bfc-8626-947aada8ecde\") " pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:25.047935 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:25.047879    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:25.047935 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:25.047901    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:25.047935 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:25.047910    2573 projected.go:196] Error preparing data for projected volume kube-api-access-9pmjv for pod openshift-network-diagnostics/network-check-target-wtdtc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:25.047935 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:25.047917    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:25.048170 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:25.047964    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv podName:fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae nodeName:}" failed. No retries permitted until 2026-07-09 17:39:33.047944185 +0000 UTC m=+17.061473292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pmjv" (UniqueName: "kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv") pod "network-check-target-wtdtc" (UID: "fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:25.048170 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:25.047980    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret podName:b282b3a8-fc56-4bfc-8626-947aada8ecde nodeName:}" failed. No retries permitted until 2026-07-09 17:39:33.047973707 +0000 UTC m=+17.061502799 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret") pod "global-pull-secret-syncer-mcsfz" (UID: "b282b3a8-fc56-4bfc-8626-947aada8ecde") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:25.522497 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:25.522456    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:25.522673 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:25.522599    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcsfz" podUID="b282b3a8-fc56-4bfc-8626-947aada8ecde"
Jul 09 17:39:26.523038 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:26.522993    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:26.523485 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:26.523100    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qvsq" podUID="8a62897b-ea47-4b2f-8762-a4fc7b7a18dc"
Jul 09 17:39:26.523485 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:26.523415    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:26.523599 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:26.523495    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtdtc" podUID="fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae"
Jul 09 17:39:27.522567 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:27.522535    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:27.522743 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:27.522649    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcsfz" podUID="b282b3a8-fc56-4bfc-8626-947aada8ecde"
Jul 09 17:39:28.522578 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:28.522545    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:28.523044 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:28.522679    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qvsq" podUID="8a62897b-ea47-4b2f-8762-a4fc7b7a18dc"
Jul 09 17:39:28.523044 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:28.522753    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:28.523044 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:28.522858    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtdtc" podUID="fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae"
Jul 09 17:39:29.522949 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:29.522919    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:29.523383 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:29.523012    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcsfz" podUID="b282b3a8-fc56-4bfc-8626-947aada8ecde"
Jul 09 17:39:30.522944 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:30.522904    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:30.523289 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:30.523248    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:30.523697 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:30.523387    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qvsq" podUID="8a62897b-ea47-4b2f-8762-a4fc7b7a18dc"
Jul 09 17:39:30.523697 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:30.523453    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtdtc" podUID="fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae"
Jul 09 17:39:31.522677 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:31.522636    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:31.522899 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:31.522773    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcsfz" podUID="b282b3a8-fc56-4bfc-8626-947aada8ecde"
Jul 09 17:39:32.523077 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:32.522908    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:32.523077 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:32.523038    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qvsq" podUID="8a62897b-ea47-4b2f-8762-a4fc7b7a18dc"
Jul 09 17:39:32.523077 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:32.523050    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:32.523475 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:32.523155    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtdtc" podUID="fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae"
Jul 09 17:39:32.701541 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:32.701507    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal" event={"ID":"0b3b8930aa1846297e408b672c61a6ab","Type":"ContainerStarted","Data":"3982af06bf6eb3483c9b1e94022b2cee548362098d3a97a5efa5129759c83ad6"}
Jul 09 17:39:32.719625 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:32.719524    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal" podStartSLOduration=15.719498485 podStartE2EDuration="15.719498485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:32.719318895 +0000 UTC m=+16.732848009" watchObservedRunningTime="2026-07-09 17:39:32.719498485 +0000 UTC m=+16.733027612"
Jul 09 17:39:33.007778 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.007358    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs\") pod \"network-metrics-daemon-6qvsq\" (UID: \"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc\") " pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:33.007882 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:33.007496    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:33.007936 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:33.007883    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs podName:8a62897b-ea47-4b2f-8762-a4fc7b7a18dc nodeName:}" failed. No retries permitted until 2026-07-09 17:39:49.007865324 +0000 UTC m=+33.021394420 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs") pod "network-metrics-daemon-6qvsq" (UID: "8a62897b-ea47-4b2f-8762-a4fc7b7a18dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:33.108621 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.108591    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret\") pod \"global-pull-secret-syncer-mcsfz\" (UID: \"b282b3a8-fc56-4bfc-8626-947aada8ecde\") " pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:33.108770 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.108640    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmjv\" (UniqueName: \"kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv\") pod \"network-check-target-wtdtc\" (UID: \"fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae\") " pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:33.108835 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:33.108766    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:33.108835 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:33.108780    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:33.108835 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:33.108794    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:33.108835 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:33.108803    2573 projected.go:196] Error preparing data for projected volume kube-api-access-9pmjv for pod openshift-network-diagnostics/network-check-target-wtdtc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:33.108835 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:33.108828    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret podName:b282b3a8-fc56-4bfc-8626-947aada8ecde nodeName:}" failed. No retries permitted until 2026-07-09 17:39:49.108809113 +0000 UTC m=+33.122338207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret") pod "global-pull-secret-syncer-mcsfz" (UID: "b282b3a8-fc56-4bfc-8626-947aada8ecde") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:33.109084 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:33.108842    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv podName:fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae nodeName:}" failed. No retries permitted until 2026-07-09 17:39:49.108836022 +0000 UTC m=+33.122365113 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pmjv" (UniqueName: "kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv") pod "network-check-target-wtdtc" (UID: "fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:33.522341 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.522263    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:33.522574 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:33.522383    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcsfz" podUID="b282b3a8-fc56-4bfc-8626-947aada8ecde"
Jul 09 17:39:33.705688 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.705649    2573 generic.go:356] "Generic (PLEG): container finished" podID="343c2c99cf08d1b78e530233805856b4" containerID="4bb6183d1680e6bc44d897c19721bef01fb810620f776b519024115fdd6de868" exitCode=0
Jul 09 17:39:33.706420 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.706184    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal" event={"ID":"343c2c99cf08d1b78e530233805856b4","Type":"ContainerDied","Data":"4bb6183d1680e6bc44d897c19721bef01fb810620f776b519024115fdd6de868"}
Jul 09 17:39:33.708162 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.708132    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cctw4" event={"ID":"a3dcc235-2a55-4e67-a42c-4d047739f8f0","Type":"ContainerStarted","Data":"55f13ab579cb7382c09c68126fc69fb462726335bd027728400a02645aa9f9e0"}
Jul 09 17:39:33.709817 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.709794    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rnsm2" event={"ID":"feb7242f-ff85-407e-a02d-edca7d9241d1","Type":"ContainerStarted","Data":"9c06782e2c9709b9b607a8b627dec1de04fb7e4c6f8c0a6f69e5661622f5e063"}
Jul 09 17:39:33.711936 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.711812    2573 generic.go:356] "Generic (PLEG): container finished" podID="6fb10245-3407-451c-a7fc-b4a6948ffb01" containerID="e25a376b48836bd12e38b049c050a1d2471c3f30418e59368ce57bbef2665171" exitCode=0
Jul 09 17:39:33.711936 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.711904    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glm8m" event={"ID":"6fb10245-3407-451c-a7fc-b4a6948ffb01","Type":"ContainerDied","Data":"e25a376b48836bd12e38b049c050a1d2471c3f30418e59368ce57bbef2665171"}
Jul 09 17:39:33.713685 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.713633    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jbqs8" event={"ID":"76f015b9-8aaa-40ee-a73e-492fb91418d0","Type":"ContainerStarted","Data":"9905505ac165431a5a3bfb3e929bea5ce2199e97f337ebe457da5ca28215dfc3"}
Jul 09 17:39:33.717513 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.717486    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/0.log"
Jul 09 17:39:33.718025 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.717937    2573 generic.go:356] "Generic (PLEG): container finished" podID="fd0f26b5-5eac-453c-ba70-2cdef4bd156d" containerID="317b35c487d28c45ba0992aff6cd7a2b97884312e1d12e4a834e980f158cee26" exitCode=1
Jul 09 17:39:33.718025 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.717975    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs" event={"ID":"fd0f26b5-5eac-453c-ba70-2cdef4bd156d","Type":"ContainerStarted","Data":"3da6492c62213fdae62dd480eebc7676643a7cfc2deb2221ecda9ba4a1893c3a"}
Jul 09 17:39:33.718160 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.718061    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs" event={"ID":"fd0f26b5-5eac-453c-ba70-2cdef4bd156d","Type":"ContainerStarted","Data":"65d87c7565f9bd27965147b8b20d12d1195d9d690f53982a089c23233f4ebf87"}
Jul 09 17:39:33.718160 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.718110    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs" event={"ID":"fd0f26b5-5eac-453c-ba70-2cdef4bd156d","Type":"ContainerStarted","Data":"bdade571da8bb1071c307b3f2f510bee0c3be1643b55d0fb6e152e8503d9f257"}
Jul 09 17:39:33.718160 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.718158    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs" event={"ID":"fd0f26b5-5eac-453c-ba70-2cdef4bd156d","Type":"ContainerStarted","Data":"de2cbbd13da731b5ff7892b996770e581faa644b89f5c065b991e4ee470395df"}
Jul 09 17:39:33.718308 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.718204    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs" event={"ID":"fd0f26b5-5eac-453c-ba70-2cdef4bd156d","Type":"ContainerDied","Data":"317b35c487d28c45ba0992aff6cd7a2b97884312e1d12e4a834e980f158cee26"}
Jul 09 17:39:33.718308 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.718252    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs" event={"ID":"fd0f26b5-5eac-453c-ba70-2cdef4bd156d","Type":"ContainerStarted","Data":"7d7c2bfaf1e36ad7ee53bf31719a79d2426177c0c05e8a2d5b31f3aa80b2bd90"}
Jul 09 17:39:33.719621 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.719583    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ps4sv" event={"ID":"26c0b0ef-78e7-455e-b5cf-52376c022784","Type":"ContainerStarted","Data":"05c44facad378020f358465cc05266c9e7a1b17a41599ddf7b9c34caa3a81c0f"}
Jul 09 17:39:33.725327 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.725280    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5" event={"ID":"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee","Type":"ContainerStarted","Data":"ef3002a3dc9ea532f88bc5fdf412ee2683050e24fcd8b53fdfdfe9d30e46ba60"}
Jul 09 17:39:33.817215 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.817119    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rnsm2" podStartSLOduration=2.977225696 podStartE2EDuration="17.817099087s" totalImagesPullingTime="14.839873391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:33.788254717 +0000 UTC m=+17.801783831" watchObservedRunningTime="2026-07-09 17:39:33.817099087 +0000 UTC m=+17.830628205"
Jul 09 17:39:33.838185 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.838144    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jbqs8" podStartSLOduration=3.052333098 podStartE2EDuration="17.838129765s" totalImagesPullingTime="14.785796667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:33.817032429 +0000 UTC m=+17.830561609" watchObservedRunningTime="2026-07-09 17:39:33.838129765 +0000 UTC m=+17.851658879"
Jul 09 17:39:33.862073 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.862023    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ps4sv" podStartSLOduration=3.014178507 podStartE2EDuration="17.8620022s" totalImagesPullingTime="14.847823693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:33.838084004 +0000 UTC m=+17.851613143" watchObservedRunningTime="2026-07-09 17:39:33.8620022 +0000 UTC m=+17.875531314"
Jul 09 17:39:33.862914 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.862873    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cctw4" podStartSLOduration=3.058165408 podStartE2EDuration="17.862860337s" totalImagesPullingTime="14.804694929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:33.862467474 +0000 UTC m=+17.875996589" watchObservedRunningTime="2026-07-09 17:39:33.862860337 +0000 UTC m=+17.876389452"
Jul 09 17:39:33.972582 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:33.972539    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 17:39:34.430646 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:34.430484    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T17:39:33.972555431Z","UUID":"7b23a849-30d8-495b-a051-0619402a5358","Handler":null,"Name":"","Endpoint":""}
Jul 09 17:39:34.432501 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:34.432478    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 17:39:34.432625 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:34.432523    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 17:39:34.522889 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:34.522864    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:34.522889 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:34.522876    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:34.523115 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:34.523005    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qvsq" podUID="8a62897b-ea47-4b2f-8762-a4fc7b7a18dc"
Jul 09 17:39:34.523115 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:34.523084    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtdtc" podUID="fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae"
Jul 09 17:39:34.730261 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:34.729942    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7gvh8" event={"ID":"fcc10c4d-ba3f-4dd7-90ca-02826f1366b9","Type":"ContainerStarted","Data":"4b4a460f8d920b9a7c06a85cb2d77c1cc8521e6e08217d510c4a83721175e70f"}
Jul 09 17:39:34.732304 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:34.732278    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5" event={"ID":"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee","Type":"ContainerStarted","Data":"fa1cddb7c0fd27a1bfc6af90059ca61ac142debd36e16dc93b9cfaa370be7f32"}
Jul 09 17:39:34.734497 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:34.734440    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal" event={"ID":"343c2c99cf08d1b78e530233805856b4","Type":"ContainerStarted","Data":"50efd7a9f6023cb2e9f5b869b6beae5718dbcb801c56ca1f99855d218e3ae161"}
Jul 09 17:39:34.749007 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:34.748962    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7gvh8" podStartSLOduration=3.894145196 podStartE2EDuration="18.748945122s" totalImagesPullingTime="14.854799926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:34.748427429 +0000 UTC m=+18.761956544" watchObservedRunningTime="2026-07-09 17:39:34.748945122 +0000 UTC m=+18.762474233"
Jul 09 17:39:34.768584 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:34.767600    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal" podStartSLOduration=17.767582657 podStartE2EDuration="17.767582657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:34.766477961 +0000 UTC m=+18.780007076" watchObservedRunningTime="2026-07-09 17:39:34.767582657 +0000 UTC m=+18.781111772"
Jul 09 17:39:35.523186 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:35.523108    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:35.523347 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:35.523227    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcsfz" podUID="b282b3a8-fc56-4bfc-8626-947aada8ecde"
Jul 09 17:39:35.741784 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:35.741755    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/0.log"
Jul 09 17:39:35.742581 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:35.742549    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs" event={"ID":"fd0f26b5-5eac-453c-ba70-2cdef4bd156d","Type":"ContainerStarted","Data":"c3c8a596a7c518b93505901dcd5e0f7699b35f547bf86234bbdae53eec73e64d"}
Jul 09 17:39:35.746000 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:35.745973    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5" event={"ID":"8203ebfb-2681-4fba-9f3d-9aa98f4ecfee","Type":"ContainerStarted","Data":"725570fd131bc445042a0f784398ce9a75b7e078b34d07a10d182ab5fff5d608"}
Jul 09 17:39:35.769693 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:35.769651    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mfh5" podStartSLOduration=3.107273246 podStartE2EDuration="19.76963656s" totalImagesPullingTime="16.662363314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:35.768858032 +0000 UTC m=+19.782387173" watchObservedRunningTime="2026-07-09 17:39:35.76963656 +0000 UTC m=+19.783165675"
Jul 09 17:39:36.162330 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:36.162296    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ps4sv"
Jul 09 17:39:36.523442 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:36.523392    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:36.523688 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:36.523513    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qvsq" podUID="8a62897b-ea47-4b2f-8762-a4fc7b7a18dc"
Jul 09 17:39:36.523688 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:36.523556    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:36.523688 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:36.523657    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtdtc" podUID="fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae"
Jul 09 17:39:36.860103 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:36.859831    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ps4sv"
Jul 09 17:39:36.860547 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:36.860504    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ps4sv"
Jul 09 17:39:37.522236 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:37.522204    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:37.522392 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:37.522342    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcsfz" podUID="b282b3a8-fc56-4bfc-8626-947aada8ecde"
Jul 09 17:39:37.753133 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:37.753089    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ps4sv"
Jul 09 17:39:38.522326 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:38.522292    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:38.522989 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:38.522336    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:38.522989 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:38.522447    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qvsq" podUID="8a62897b-ea47-4b2f-8762-a4fc7b7a18dc"
Jul 09 17:39:38.522989 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:38.522591    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtdtc" podUID="fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae"
Jul 09 17:39:38.757073 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:38.757049    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/0.log"
Jul 09 17:39:38.757537 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:38.757456    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs" event={"ID":"fd0f26b5-5eac-453c-ba70-2cdef4bd156d","Type":"ContainerStarted","Data":"5267a9a4291f9a1bd9e9da9e8b0dcdb1371608295e64d0bc29d4317e19340ba4"}
Jul 09 17:39:38.757739 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:38.757698    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:38.757865 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:38.757788    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:38.758036 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:38.758018    2573 scope.go:117] "RemoveContainer" containerID="317b35c487d28c45ba0992aff6cd7a2b97884312e1d12e4a834e980f158cee26"
Jul 09 17:39:38.760356 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:38.760335    2573 generic.go:356] "Generic (PLEG): container finished" podID="6fb10245-3407-451c-a7fc-b4a6948ffb01" containerID="4eb3b7d656b7ff415571b9060c81906b83e6f106e1ea3adaefe8438c3f87533c" exitCode=0
Jul 09 17:39:38.760705 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:38.760680    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glm8m" event={"ID":"6fb10245-3407-451c-a7fc-b4a6948ffb01","Type":"ContainerDied","Data":"4eb3b7d656b7ff415571b9060c81906b83e6f106e1ea3adaefe8438c3f87533c"}
Jul 09 17:39:38.775447 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:38.775425    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:39.522804 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:39.522757    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:39.523520 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:39.522881    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcsfz" podUID="b282b3a8-fc56-4bfc-8626-947aada8ecde"
Jul 09 17:39:40.257262 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:40.257235    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wtdtc"]
Jul 09 17:39:40.257436 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:40.257381    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:40.257499 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:40.257480    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtdtc" podUID="fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae"
Jul 09 17:39:40.260336 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:40.260254    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6qvsq"]
Jul 09 17:39:40.260475 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:40.260461    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:40.260704 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:40.260662    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qvsq" podUID="8a62897b-ea47-4b2f-8762-a4fc7b7a18dc"
Jul 09 17:39:40.261118 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:40.261081    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mcsfz"]
Jul 09 17:39:40.261228 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:40.261172    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:40.261309 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:40.261250    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcsfz" podUID="b282b3a8-fc56-4bfc-8626-947aada8ecde"
Jul 09 17:39:41.523042 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:41.523015    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:41.523395 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:41.523049    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:41.523395 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:41.523140    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtdtc" podUID="fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae"
Jul 09 17:39:41.523395 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:41.523282    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcsfz" podUID="b282b3a8-fc56-4bfc-8626-947aada8ecde"
Jul 09 17:39:42.522416 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:42.522391    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:42.522548 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:42.522525    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qvsq" podUID="8a62897b-ea47-4b2f-8762-a4fc7b7a18dc"
Jul 09 17:39:42.770646 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:42.770398    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ds27w" event={"ID":"46f1c9d1-6578-4343-88ba-900a5cc5bd28","Type":"ContainerStarted","Data":"d3c770045ce83c5f7f4da1a64d9b25115f813fc2a051025c269749b5ca3a3551"}
Jul 09 17:39:42.774416 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:42.774390    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/0.log"
Jul 09 17:39:42.774825 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:42.774796    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs" event={"ID":"fd0f26b5-5eac-453c-ba70-2cdef4bd156d","Type":"ContainerStarted","Data":"ee1db8b29ed0e69d32b5535744061ec96483699c7eee5607b00c820e9b2ee0fc"}
Jul 09 17:39:42.775183 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:42.775159    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:42.791989 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:42.791957    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:42.823428 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:42.823378    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ds27w" podStartSLOduration=1.8703272389999999 podStartE2EDuration="26.823359369s" totalImagesPullingTime="24.95303213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-09 17:39:42.790594303 +0000 UTC m=+26.804123430" watchObservedRunningTime="2026-07-09 17:39:42.823359369 +0000 UTC m=+26.836888484"
Jul 09 17:39:42.854565 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:42.854510    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs" podStartSLOduration=11.907221578 podStartE2EDuration="26.854492025s" totalImagesPullingTime="14.947270447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:42.824229847 +0000 UTC m=+26.837758960" watchObservedRunningTime="2026-07-09 17:39:42.854492025 +0000 UTC m=+26.868021142"
Jul 09 17:39:43.522790 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:43.522763    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:43.522896 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:43.522795    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:43.522954 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:43.522883    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcsfz" podUID="b282b3a8-fc56-4bfc-8626-947aada8ecde"
Jul 09 17:39:43.523007 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:43.522990    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtdtc" podUID="fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae"
Jul 09 17:39:43.778815 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:43.778781    2573 generic.go:356] "Generic (PLEG): container finished" podID="6fb10245-3407-451c-a7fc-b4a6948ffb01" containerID="4a3e9fcb39679810ad89345e057e8b94cf88bb1cd282f463c7ba592257224f04" exitCode=0
Jul 09 17:39:43.779503 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:43.778861    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glm8m" event={"ID":"6fb10245-3407-451c-a7fc-b4a6948ffb01","Type":"ContainerDied","Data":"4a3e9fcb39679810ad89345e057e8b94cf88bb1cd282f463c7ba592257224f04"}
Jul 09 17:39:43.791308 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:43.791283    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-64kgs"
Jul 09 17:39:44.523069 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:44.523037    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:44.523258 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:44.523171    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qvsq" podUID="8a62897b-ea47-4b2f-8762-a4fc7b7a18dc"
Jul 09 17:39:45.263939 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.263910    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeReady"
Jul 09 17:39:45.264397 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.264017    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 17:39:45.323124 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.323097    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-558c646897-dqv7k"]
Jul 09 17:39:45.346031 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.346011    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-675dp"]
Jul 09 17:39:45.346159 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.346142    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.351134 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.351108    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-48n2x\""
Jul 09 17:39:45.351761 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.351714    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 17:39:45.351761 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.351747    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 17:39:45.351925 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.351791    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 17:39:45.360377 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.360359    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:45.362204 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.362188    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 17:39:45.365733 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.365644    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 17:39:45.366295 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.366045    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2k2gp\""
Jul 09 17:39:45.366360 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.366317    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 17:39:45.368063 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.368046    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-558c646897-dqv7k"]
Jul 09 17:39:45.368589 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.368567    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9rw5k"]
Jul 09 17:39:45.386182 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.386161    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9rw5k"
Jul 09 17:39:45.386182 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.386162    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-675dp"]
Jul 09 17:39:45.388421 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.388390    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9rw5k"]
Jul 09 17:39:45.389565 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.389544    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 17:39:45.389565 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.389564    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 17:39:45.389683 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.389546    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 17:39:45.389878 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.389865    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-scpst\""
Jul 09 17:39:45.503538 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.503506    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517-cert\") pod \"ingress-canary-9rw5k\" (UID: \"00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517\") " pod="openshift-ingress-canary/ingress-canary-9rw5k"
Jul 09 17:39:45.503700 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.503575    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-certificates\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.503700 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.503610    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e765850-1ceb-4508-a2d8-876bf7fd3a69-ca-trust-extracted\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.503700 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.503636    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j227v\" (UniqueName: \"kubernetes.io/projected/00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517-kube-api-access-j227v\") pod \"ingress-canary-9rw5k\" (UID: \"00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517\") " pod="openshift-ingress-canary/ingress-canary-9rw5k"
Jul 09 17:39:45.503700 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.503659    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e765850-1ceb-4508-a2d8-876bf7fd3a69-installation-pull-secrets\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.503700 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.503694    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8l4p\" (UniqueName: \"kubernetes.io/projected/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-kube-api-access-n8l4p\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:45.503995 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.503740    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-bound-sa-token\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.503995 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.503765    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:45.503995 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.503790    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e765850-1ceb-4508-a2d8-876bf7fd3a69-trusted-ca\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.503995 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.503818    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-tmp-dir\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:45.503995 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.503846    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.503995 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.503885    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e765850-1ceb-4508-a2d8-876bf7fd3a69-image-registry-private-configuration\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.503995 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.503911    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8brwq\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-kube-api-access-8brwq\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.503995 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.503978    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-config-volume\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:45.522640 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.522582    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:45.522640 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.522614    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:45.525953 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.525932    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 17:39:45.526063 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.525976    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-59446\""
Jul 09 17:39:45.526442 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.526375    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 17:39:45.526529 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.526444    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 17:39:45.603497 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.603475    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ps4sv_26c0b0ef-78e7-455e-b5cf-52376c022784/konnectivity-agent/0.log" limit=1024
Jul 09 17:39:45.604365 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.604345    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517-cert\") pod \"ingress-canary-9rw5k\" (UID: \"00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517\") " pod="openshift-ingress-canary/ingress-canary-9rw5k"
Jul 09 17:39:45.604441 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.604389    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-certificates\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.604441 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.604418    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e765850-1ceb-4508-a2d8-876bf7fd3a69-ca-trust-extracted\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.604541 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.604443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j227v\" (UniqueName: \"kubernetes.io/projected/00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517-kube-api-access-j227v\") pod \"ingress-canary-9rw5k\" (UID: \"00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517\") " pod="openshift-ingress-canary/ingress-canary-9rw5k"
Jul 09 17:39:45.604541 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.604467    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e765850-1ceb-4508-a2d8-876bf7fd3a69-installation-pull-secrets\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.604541 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.604494    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8l4p\" (UniqueName: \"kubernetes.io/projected/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-kube-api-access-n8l4p\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:45.604541 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.604519    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-bound-sa-token\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.604735 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.604675    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:45.604735 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.604701    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e765850-1ceb-4508-a2d8-876bf7fd3a69-trusted-ca\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.604825 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.604752    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-tmp-dir\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:45.604825 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:45.604760    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:39:45.604825 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.604782    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.604825 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:45.604801    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls podName:c71f2394-9aa0-40bf-a07c-26cd48a93a1c nodeName:}" failed. No retries permitted until 2026-07-09 17:39:46.104789255 +0000 UTC m=+30.118318346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls") pod "dns-default-675dp" (UID: "c71f2394-9aa0-40bf-a07c-26cd48a93a1c") : secret "dns-default-metrics-tls" not found
Jul 09 17:39:45.605002 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:45.604852    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 17:39:45.605002 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:45.604865    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558c646897-dqv7k: secret "image-registry-tls" not found
Jul 09 17:39:45.605002 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:45.604912    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls podName:2e765850-1ceb-4508-a2d8-876bf7fd3a69 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:46.104895618 +0000 UTC m=+30.118424929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls") pod "image-registry-558c646897-dqv7k" (UID: "2e765850-1ceb-4508-a2d8-876bf7fd3a69") : secret "image-registry-tls" not found
Jul 09 17:39:45.605002 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.604862    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e765850-1ceb-4508-a2d8-876bf7fd3a69-image-registry-private-configuration\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.605002 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.604972    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8brwq\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-kube-api-access-8brwq\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.605190 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.605006    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-config-volume\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:45.605190 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.605012    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-certificates\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.605190 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.605152    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-tmp-dir\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:45.605436 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.605420    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-config-volume\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:45.605581 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.605564    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e765850-1ceb-4508-a2d8-876bf7fd3a69-trusted-ca\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.608876 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.608858    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517-cert\") pod \"ingress-canary-9rw5k\" (UID: \"00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517\") " pod="openshift-ingress-canary/ingress-canary-9rw5k"
Jul 09 17:39:45.608969 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.608911    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e765850-1ceb-4508-a2d8-876bf7fd3a69-ca-trust-extracted\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.610617 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.610599    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e765850-1ceb-4508-a2d8-876bf7fd3a69-installation-pull-secrets\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.610703 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.610638    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e765850-1ceb-4508-a2d8-876bf7fd3a69-image-registry-private-configuration\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.613602 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.613582    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j227v\" (UniqueName: \"kubernetes.io/projected/00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517-kube-api-access-j227v\") pod \"ingress-canary-9rw5k\" (UID: \"00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517\") " pod="openshift-ingress-canary/ingress-canary-9rw5k"
Jul 09 17:39:45.614771 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.614754    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-bound-sa-token\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.615261 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.615240    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8brwq\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-kube-api-access-8brwq\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:45.615982 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.615967    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8l4p\" (UniqueName: \"kubernetes.io/projected/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-kube-api-access-n8l4p\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:45.694844 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.694823    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9rw5k"
Jul 09 17:39:45.708154 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.708100    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ps4sv_26c0b0ef-78e7-455e-b5cf-52376c022784/konnectivity-agent/0.log" limit=1024
Jul 09 17:39:45.786247 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.786161    2573 generic.go:356] "Generic (PLEG): container finished" podID="6fb10245-3407-451c-a7fc-b4a6948ffb01" containerID="a2adeea94d4844ce833d7b6d5db7b23f07ac55f1b654f666d5676ce05ed99dc7" exitCode=0
Jul 09 17:39:45.786363 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.786284    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glm8m" event={"ID":"6fb10245-3407-451c-a7fc-b4a6948ffb01","Type":"ContainerDied","Data":"a2adeea94d4844ce833d7b6d5db7b23f07ac55f1b654f666d5676ce05ed99dc7"}
Jul 09 17:39:45.866206 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:45.866055    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9rw5k"]
Jul 09 17:39:45.869410 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:45.869387    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a8e0bf_d0bf_4c7b_9f33_d7dde97ba517.slice/crio-99c889a6d4fa68a504943e6b566a857a505a4914905f7dee9651714acae09696 WatchSource:0}: Error finding container 99c889a6d4fa68a504943e6b566a857a505a4914905f7dee9651714acae09696: Status 404 returned error can't find the container with id 99c889a6d4fa68a504943e6b566a857a505a4914905f7dee9651714acae09696
Jul 09 17:39:46.109819 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:46.109739    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:46.109819 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:46.109776    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:46.109956 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:46.109882    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:39:46.109956 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:46.109918    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 17:39:46.109956 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:46.109932    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558c646897-dqv7k: secret "image-registry-tls" not found
Jul 09 17:39:46.109956 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:46.109945    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls podName:c71f2394-9aa0-40bf-a07c-26cd48a93a1c nodeName:}" failed. No retries permitted until 2026-07-09 17:39:47.109927482 +0000 UTC m=+31.123456577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls") pod "dns-default-675dp" (UID: "c71f2394-9aa0-40bf-a07c-26cd48a93a1c") : secret "dns-default-metrics-tls" not found
Jul 09 17:39:46.110075 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:46.109963    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls podName:2e765850-1ceb-4508-a2d8-876bf7fd3a69 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:47.109953183 +0000 UTC m=+31.123482293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls") pod "image-registry-558c646897-dqv7k" (UID: "2e765850-1ceb-4508-a2d8-876bf7fd3a69") : secret "image-registry-tls" not found
Jul 09 17:39:46.523308 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:46.523280    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:46.526611 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:46.526587    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 17:39:46.526766 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:46.526696    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7kn9v\""
Jul 09 17:39:46.790927 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:46.790821    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9rw5k" event={"ID":"00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517","Type":"ContainerStarted","Data":"99c889a6d4fa68a504943e6b566a857a505a4914905f7dee9651714acae09696"}
Jul 09 17:39:47.118035 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:47.117949    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:47.118035 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:47.117996    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:47.118281 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:47.118128    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:39:47.118281 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:47.118151    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 17:39:47.118281 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:47.118167    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558c646897-dqv7k: secret "image-registry-tls" not found
Jul 09 17:39:47.118281 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:47.118201    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls podName:c71f2394-9aa0-40bf-a07c-26cd48a93a1c nodeName:}" failed. No retries permitted until 2026-07-09 17:39:49.118179636 +0000 UTC m=+33.131708728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls") pod "dns-default-675dp" (UID: "c71f2394-9aa0-40bf-a07c-26cd48a93a1c") : secret "dns-default-metrics-tls" not found
Jul 09 17:39:47.118281 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:47.118229    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls podName:2e765850-1ceb-4508-a2d8-876bf7fd3a69 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:49.118210377 +0000 UTC m=+33.131739486 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls") pod "image-registry-558c646897-dqv7k" (UID: "2e765850-1ceb-4508-a2d8-876bf7fd3a69") : secret "image-registry-tls" not found
Jul 09 17:39:48.798827 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:48.798778    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9rw5k" event={"ID":"00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517","Type":"ContainerStarted","Data":"4dd2d1dae50705a06e6cb9e0dfcb8277fe67242f552c34c9e8864f6b312aab63"}
Jul 09 17:39:48.818838 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:48.818796    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9rw5k" podStartSLOduration=1.216742258 podStartE2EDuration="3.818777103s" totalImagesPullingTime="2.602034845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:48.818020414 +0000 UTC m=+32.831549551" watchObservedRunningTime="2026-07-09 17:39:48.818777103 +0000 UTC m=+32.832306216"
Jul 09 17:39:49.031114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:49.031068    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs\") pod \"network-metrics-daemon-6qvsq\" (UID: \"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc\") " pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:39:49.031297 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:49.031242    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 17:39:49.031354 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:49.031319    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs podName:8a62897b-ea47-4b2f-8762-a4fc7b7a18dc nodeName:}" failed. No retries permitted until 2026-07-09 17:40:21.031298187 +0000 UTC m=+65.044827292 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs") pod "network-metrics-daemon-6qvsq" (UID: "8a62897b-ea47-4b2f-8762-a4fc7b7a18dc") : secret "metrics-daemon-secret" not found
Jul 09 17:39:49.131900 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:49.131809    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmjv\" (UniqueName: \"kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv\") pod \"network-check-target-wtdtc\" (UID: \"fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae\") " pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:49.131900 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:49.131872    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:49.132122 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:49.131913    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:49.132122 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:49.131974    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret\") pod \"global-pull-secret-syncer-mcsfz\" (UID: \"b282b3a8-fc56-4bfc-8626-947aada8ecde\") " pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:49.132122 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:49.132017    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:39:49.132122 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:49.132063    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 17:39:49.132122 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:49.132081    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558c646897-dqv7k: secret "image-registry-tls" not found
Jul 09 17:39:49.132122 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:49.132090    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls podName:c71f2394-9aa0-40bf-a07c-26cd48a93a1c nodeName:}" failed. No retries permitted until 2026-07-09 17:39:53.132069912 +0000 UTC m=+37.145599010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls") pod "dns-default-675dp" (UID: "c71f2394-9aa0-40bf-a07c-26cd48a93a1c") : secret "dns-default-metrics-tls" not found
Jul 09 17:39:49.132397 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:49.132130    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls podName:2e765850-1ceb-4508-a2d8-876bf7fd3a69 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:53.132113727 +0000 UTC m=+37.145642839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls") pod "image-registry-558c646897-dqv7k" (UID: "2e765850-1ceb-4508-a2d8-876bf7fd3a69") : secret "image-registry-tls" not found
Jul 09 17:39:49.136247 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:49.136220    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b282b3a8-fc56-4bfc-8626-947aada8ecde-original-pull-secret\") pod \"global-pull-secret-syncer-mcsfz\" (UID: \"b282b3a8-fc56-4bfc-8626-947aada8ecde\") " pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:49.136498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:49.136473    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pmjv\" (UniqueName: \"kubernetes.io/projected/fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae-kube-api-access-9pmjv\") pod \"network-check-target-wtdtc\" (UID: \"fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae\") " pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:49.138640 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:49.138445    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:49.434751 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:49.434652    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcsfz"
Jul 09 17:39:50.590048 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:50.589978    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mcsfz"]
Jul 09 17:39:50.592553 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:50.592528    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb282b3a8_fc56_4bfc_8626_947aada8ecde.slice/crio-f5557ae1b34106b91708acc7605d5c1e03c4c207cf2d69bae791f9a1258aca53 WatchSource:0}: Error finding container f5557ae1b34106b91708acc7605d5c1e03c4c207cf2d69bae791f9a1258aca53: Status 404 returned error can't find the container with id f5557ae1b34106b91708acc7605d5c1e03c4c207cf2d69bae791f9a1258aca53
Jul 09 17:39:50.593294 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:50.593275    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wtdtc"]
Jul 09 17:39:50.598011 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:39:50.597982    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0f43af_cdd0_41f4_bedb_ba61bdfd16ae.slice/crio-1406050f3fbe9253478635cc5ef46699f730baef075ecf6a8b4e61c6b2f57c4d WatchSource:0}: Error finding container 1406050f3fbe9253478635cc5ef46699f730baef075ecf6a8b4e61c6b2f57c4d: Status 404 returned error can't find the container with id 1406050f3fbe9253478635cc5ef46699f730baef075ecf6a8b4e61c6b2f57c4d
Jul 09 17:39:50.804610 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:50.804579    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mcsfz" event={"ID":"b282b3a8-fc56-4bfc-8626-947aada8ecde","Type":"ContainerStarted","Data":"f5557ae1b34106b91708acc7605d5c1e03c4c207cf2d69bae791f9a1258aca53"}
Jul 09 17:39:50.805635 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:50.805612    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wtdtc" event={"ID":"fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae","Type":"ContainerStarted","Data":"1406050f3fbe9253478635cc5ef46699f730baef075ecf6a8b4e61c6b2f57c4d"}
Jul 09 17:39:51.812958 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:51.812845    2573 generic.go:356] "Generic (PLEG): container finished" podID="6fb10245-3407-451c-a7fc-b4a6948ffb01" containerID="f2d606246f8af31ec5566e3c4475a5f7d297b212a4722657d44ebc20b83bc3d9" exitCode=0
Jul 09 17:39:51.812958 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:51.812965    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glm8m" event={"ID":"6fb10245-3407-451c-a7fc-b4a6948ffb01","Type":"ContainerDied","Data":"f2d606246f8af31ec5566e3c4475a5f7d297b212a4722657d44ebc20b83bc3d9"}
Jul 09 17:39:52.819937 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:52.819895    2573 generic.go:356] "Generic (PLEG): container finished" podID="6fb10245-3407-451c-a7fc-b4a6948ffb01" containerID="4370f7f747321c32ea056ea21e36c671c4f7002513d9c6e7e73dd80f2808fd95" exitCode=0
Jul 09 17:39:52.820463 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:52.819981    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glm8m" event={"ID":"6fb10245-3407-451c-a7fc-b4a6948ffb01","Type":"ContainerDied","Data":"4370f7f747321c32ea056ea21e36c671c4f7002513d9c6e7e73dd80f2808fd95"}
Jul 09 17:39:53.168082 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:53.167990    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:39:53.168082 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:53.168041    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:39:53.168293 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:53.168159    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 17:39:53.168293 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:53.168164    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:39:53.168293 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:53.168235    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls podName:c71f2394-9aa0-40bf-a07c-26cd48a93a1c nodeName:}" failed. No retries permitted until 2026-07-09 17:40:01.168214403 +0000 UTC m=+45.181743509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls") pod "dns-default-675dp" (UID: "c71f2394-9aa0-40bf-a07c-26cd48a93a1c") : secret "dns-default-metrics-tls" not found
Jul 09 17:39:53.168293 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:53.168176    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558c646897-dqv7k: secret "image-registry-tls" not found
Jul 09 17:39:53.168487 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:39:53.168315    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls podName:2e765850-1ceb-4508-a2d8-876bf7fd3a69 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:01.168296357 +0000 UTC m=+45.181825464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls") pod "image-registry-558c646897-dqv7k" (UID: "2e765850-1ceb-4508-a2d8-876bf7fd3a69") : secret "image-registry-tls" not found
Jul 09 17:39:53.826152 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:53.826118    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-glm8m" event={"ID":"6fb10245-3407-451c-a7fc-b4a6948ffb01","Type":"ContainerStarted","Data":"87ca4cf6f62acc557e8b813e97fd5015f016439108f32e1484025b11db932ae5"}
Jul 09 17:39:53.856578 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:53.856524    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-glm8m" podStartSLOduration=7.6351262779999995 podStartE2EDuration="37.856503395s" totalImagesPullingTime="30.221377117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:53.854559431 +0000 UTC m=+37.868088548" watchObservedRunningTime="2026-07-09 17:39:53.856503395 +0000 UTC m=+37.870032509"
Jul 09 17:39:56.833131 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:56.833093    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mcsfz" event={"ID":"b282b3a8-fc56-4bfc-8626-947aada8ecde","Type":"ContainerStarted","Data":"dfa84fac4e284b3097010bbd4b75f322665433ce109fa19e214172b4e8e9864b"}
Jul 09 17:39:56.834451 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:56.834428    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wtdtc" event={"ID":"fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae","Type":"ContainerStarted","Data":"50426b49dbdc1b3d57549495d87f311173b7dc008c463bfde3817fda4aa1203f"}
Jul 09 17:39:56.834575 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:56.834558    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:39:56.850482 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:39:56.850444    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mcsfz" podStartSLOduration=35.328736628 podStartE2EDuration="40.85043294s" totalImagesPullingTime="5.521696312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:56.850001975 +0000 UTC m=+40.863531089" watchObservedRunningTime="2026-07-09 17:39:56.85043294 +0000 UTC m=+40.863962054"
Jul 09 17:40:00.695027 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.694971    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wtdtc" podStartSLOduration=39.188469612 podStartE2EDuration="44.694954205s" totalImagesPullingTime="5.506484593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:56.867039351 +0000 UTC m=+40.880568464" watchObservedRunningTime="2026-07-09 17:40:00.694954205 +0000 UTC m=+44.708483344"
Jul 09 17:40:00.695895 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.695871    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"]
Jul 09 17:40:00.708132 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.708110    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"]
Jul 09 17:40:00.708247 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.708209    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:00.712048 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.712032    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 09 17:40:00.712048 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.712041    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 17:40:00.714113 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.714093    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 17:40:00.714197 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.714098    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 09 17:40:00.715336 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.715318    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hjrxj\""
Jul 09 17:40:00.803540 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.803517    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-dkxlh"]
Jul 09 17:40:00.819090 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.819067    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wp6zp\" (UID: \"de82e36c-3a74-425b-b950-6983d4686d0a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:00.819186 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.819109    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc2wl\" (UniqueName: \"kubernetes.io/projected/de82e36c-3a74-425b-b950-6983d4686d0a-kube-api-access-cc2wl\") pod \"cluster-monitoring-operator-5f7f494dd4-wp6zp\" (UID: \"de82e36c-3a74-425b-b950-6983d4686d0a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:00.819255 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.819239    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/de82e36c-3a74-425b-b950-6983d4686d0a-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-wp6zp\" (UID: \"de82e36c-3a74-425b-b950-6983d4686d0a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:00.824253 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.824239    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dkxlh"
Jul 09 17:40:00.827519 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.827501    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs"]
Jul 09 17:40:00.828628 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.828145    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-x7zdz\""
Jul 09 17:40:00.844304 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.844287    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-xkn6j"]
Jul 09 17:40:00.844415 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.844400    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs"
Jul 09 17:40:00.851005 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.850988    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 09 17:40:00.854965 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.854943    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 09 17:40:00.855054 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.855003    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:40:00.855054 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.855013    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 09 17:40:00.855238 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.855225    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-snmz9\""
Jul 09 17:40:00.864394 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.864376    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs"]
Jul 09 17:40:00.864471 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.864450    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:00.867423 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.867400    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 17:40:00.867507 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.867450    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 09 17:40:00.867507 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.867456    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 09 17:40:00.869659 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.869642    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 17:40:00.870534 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.870514    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-x2tl8\""
Jul 09 17:40:00.878395 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.878375    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 09 17:40:00.888870 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.888851    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"]
Jul 09 17:40:00.904446 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.904429    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-dkxlh"]
Jul 09 17:40:00.904525 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.904499    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"
Jul 09 17:40:00.908614 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.908593    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:40:00.908695 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.908624    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 09 17:40:00.909198 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.908989    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cd8xx\""
Jul 09 17:40:00.909198 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.909015    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 09 17:40:00.909966 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.909947    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"]
Jul 09 17:40:00.916382 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.916364    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lqj4g"]
Jul 09 17:40:00.919917 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.919894    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9hnt\" (UniqueName: \"kubernetes.io/projected/04d0d87c-7e14-4ad2-8e16-b125198c2820-kube-api-access-k9hnt\") pod \"network-check-source-869dcfcdcd-dkxlh\" (UID: \"04d0d87c-7e14-4ad2-8e16-b125198c2820\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dkxlh"
Jul 09 17:40:00.920003 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.919983    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/de82e36c-3a74-425b-b950-6983d4686d0a-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-wp6zp\" (UID: \"de82e36c-3a74-425b-b950-6983d4686d0a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:00.920052 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.920034    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wp6zp\" (UID: \"de82e36c-3a74-425b-b950-6983d4686d0a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:00.920149 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:00.920135    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 17:40:00.920195 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.920157    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc2wl\" (UniqueName: \"kubernetes.io/projected/de82e36c-3a74-425b-b950-6983d4686d0a-kube-api-access-cc2wl\") pod \"cluster-monitoring-operator-5f7f494dd4-wp6zp\" (UID: \"de82e36c-3a74-425b-b950-6983d4686d0a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:00.920246 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:00.920199    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls podName:de82e36c-3a74-425b-b950-6983d4686d0a nodeName:}" failed. No retries permitted until 2026-07-09 17:40:01.420183447 +0000 UTC m=+45.433712551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-wp6zp" (UID: "de82e36c-3a74-425b-b950-6983d4686d0a") : secret "cluster-monitoring-operator-tls" not found
Jul 09 17:40:00.920742 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.920709    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/de82e36c-3a74-425b-b950-6983d4686d0a-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-wp6zp\" (UID: \"de82e36c-3a74-425b-b950-6983d4686d0a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:00.940682 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.940664    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lqj4g"
Jul 09 17:40:00.943278 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.943261    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lqj4g"]
Jul 09 17:40:00.946590 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.946530    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 09 17:40:00.947440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.946883    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9shmx\""
Jul 09 17:40:00.947440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.947165    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:40:00.951617 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.951594    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc2wl\" (UniqueName: \"kubernetes.io/projected/de82e36c-3a74-425b-b950-6983d4686d0a-kube-api-access-cc2wl\") pod \"cluster-monitoring-operator-5f7f494dd4-wp6zp\" (UID: \"de82e36c-3a74-425b-b950-6983d4686d0a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:00.952293 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:00.952273    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-xkn6j"]
Jul 09 17:40:01.004585 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.004561    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5"]
Jul 09 17:40:01.021434 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.021409    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9hnt\" (UniqueName: \"kubernetes.io/projected/04d0d87c-7e14-4ad2-8e16-b125198c2820-kube-api-access-k9hnt\") pod \"network-check-source-869dcfcdcd-dkxlh\" (UID: \"04d0d87c-7e14-4ad2-8e16-b125198c2820\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dkxlh"
Jul 09 17:40:01.021535 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.021449    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-snapshots\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.021535 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.021473    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.021641 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.021546    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00d27200-da1b-4d8a-9e38-65676147b344-serving-cert\") pod \"service-ca-operator-7695fc6c8c-rlwgs\" (UID: \"00d27200-da1b-4d8a-9e38-65676147b344\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs"
Jul 09 17:40:01.021641 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.021577    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00d27200-da1b-4d8a-9e38-65676147b344-config\") pod \"service-ca-operator-7695fc6c8c-rlwgs\" (UID: \"00d27200-da1b-4d8a-9e38-65676147b344\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs"
Jul 09 17:40:01.021641 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.021602    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pqdl\" (UniqueName: \"kubernetes.io/projected/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-kube-api-access-7pqdl\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.021641 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.021625    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-tmp\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.021815 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.021711    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-serving-cert\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.021815 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.021750    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wgnvc\" (UID: \"45efc6ae-3d06-4f26-bb74-af2c6f8e7e97\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"
Jul 09 17:40:01.021815 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.021769    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b62h4\" (UniqueName: \"kubernetes.io/projected/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-kube-api-access-b62h4\") pod \"cluster-samples-operator-65db858894-wgnvc\" (UID: \"45efc6ae-3d06-4f26-bb74-af2c6f8e7e97\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"
Jul 09 17:40:01.021815 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.021804    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.021939 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.021862    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79pdm\" (UniqueName: \"kubernetes.io/projected/00d27200-da1b-4d8a-9e38-65676147b344-kube-api-access-79pdm\") pod \"service-ca-operator-7695fc6c8c-rlwgs\" (UID: \"00d27200-da1b-4d8a-9e38-65676147b344\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs"
Jul 09 17:40:01.028634 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.028617    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-mj9vb"]
Jul 09 17:40:01.028756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.028739    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5"
Jul 09 17:40:01.031950 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.031925    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 09 17:40:01.032049 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.032009    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jtjtb\""
Jul 09 17:40:01.032233 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.032218    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:40:01.032543 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.032525    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 09 17:40:01.032634 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.032565    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 09 17:40:01.043092 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.043026    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9hnt\" (UniqueName: \"kubernetes.io/projected/04d0d87c-7e14-4ad2-8e16-b125198c2820-kube-api-access-k9hnt\") pod \"network-check-source-869dcfcdcd-dkxlh\" (UID: \"04d0d87c-7e14-4ad2-8e16-b125198c2820\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dkxlh"
Jul 09 17:40:01.047474 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.047458    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-d6b67c788-g98z2"]
Jul 09 17:40:01.047571 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.047556    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:01.050891 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.050873    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 09 17:40:01.051420 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.051407    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:40:01.051634 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.051619    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7tdtr\""
Jul 09 17:40:01.051681 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.051619    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 09 17:40:01.053668 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.053646    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 09 17:40:01.060288 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.060270    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 09 17:40:01.069759 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.069743    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5"]
Jul 09 17:40:01.069818 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.069772    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-mj9vb"]
Jul 09 17:40:01.069818 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.069786    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d6b67c788-g98z2"]
Jul 09 17:40:01.069818 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.069789    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.072933 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.072914    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 09 17:40:01.073217 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.073197    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 09 17:40:01.073315 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.073217    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 09 17:40:01.073315 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.073225    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 09 17:40:01.073315 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.073289    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-msg4q\""
Jul 09 17:40:01.073478 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.073432    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 09 17:40:01.073575 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.073560    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 09 17:40:01.122679 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.122652    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00d27200-da1b-4d8a-9e38-65676147b344-serving-cert\") pod \"service-ca-operator-7695fc6c8c-rlwgs\" (UID: \"00d27200-da1b-4d8a-9e38-65676147b344\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs"
Jul 09 17:40:01.122832 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.122685    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00d27200-da1b-4d8a-9e38-65676147b344-config\") pod \"service-ca-operator-7695fc6c8c-rlwgs\" (UID: \"00d27200-da1b-4d8a-9e38-65676147b344\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs"
Jul 09 17:40:01.122832 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.122714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pqdl\" (UniqueName: \"kubernetes.io/projected/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-kube-api-access-7pqdl\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.122832 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.122747    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-tmp\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.122832 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.122776    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8slz\" (UniqueName: \"kubernetes.io/projected/4faab181-b8e3-4a28-bc8f-0c59f5e55edd-kube-api-access-n8slz\") pod \"volume-data-source-validator-78495597f6-lqj4g\" (UID: \"4faab181-b8e3-4a28-bc8f-0c59f5e55edd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lqj4g"
Jul 09 17:40:01.122832 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.122803    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-serving-cert\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.122832 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.122818    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wgnvc\" (UID: \"45efc6ae-3d06-4f26-bb74-af2c6f8e7e97\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"
Jul 09 17:40:01.123169 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.122864    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b62h4\" (UniqueName: \"kubernetes.io/projected/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-kube-api-access-b62h4\") pod \"cluster-samples-operator-65db858894-wgnvc\" (UID: \"45efc6ae-3d06-4f26-bb74-af2c6f8e7e97\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"
Jul 09 17:40:01.123169 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.122905    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 17:40:01.123169 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.122969    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls podName:45efc6ae-3d06-4f26-bb74-af2c6f8e7e97 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:01.622952192 +0000 UTC m=+45.636481304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls") pod "cluster-samples-operator-65db858894-wgnvc" (UID: "45efc6ae-3d06-4f26-bb74-af2c6f8e7e97") : secret "samples-operator-tls" not found
Jul 09 17:40:01.123169 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.122906    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.123169 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.123045    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79pdm\" (UniqueName: \"kubernetes.io/projected/00d27200-da1b-4d8a-9e38-65676147b344-kube-api-access-79pdm\") pod \"service-ca-operator-7695fc6c8c-rlwgs\" (UID: \"00d27200-da1b-4d8a-9e38-65676147b344\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs"
Jul 09 17:40:01.123169 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.123087    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-snapshots\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.123169 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.123113    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.123479 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.123207    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-tmp\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.123479 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.123331    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00d27200-da1b-4d8a-9e38-65676147b344-config\") pod \"service-ca-operator-7695fc6c8c-rlwgs\" (UID: \"00d27200-da1b-4d8a-9e38-65676147b344\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs"
Jul 09 17:40:01.123479 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.123474    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.123771 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.123751    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.123771 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.123774    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-snapshots\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.125181 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.125163    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-serving-cert\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.125491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.125478    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00d27200-da1b-4d8a-9e38-65676147b344-serving-cert\") pod \"service-ca-operator-7695fc6c8c-rlwgs\" (UID: \"00d27200-da1b-4d8a-9e38-65676147b344\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs"
Jul 09 17:40:01.133615 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.133600    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dkxlh"
Jul 09 17:40:01.138533 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.138515    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pqdl\" (UniqueName: \"kubernetes.io/projected/1a2bee9d-5d0f-4c62-a69b-b8e7993d858a-kube-api-access-7pqdl\") pod \"insights-operator-68d9dd94cf-xkn6j\" (UID: \"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a\") " pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.141948 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.141925    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b62h4\" (UniqueName: \"kubernetes.io/projected/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-kube-api-access-b62h4\") pod \"cluster-samples-operator-65db858894-wgnvc\" (UID: \"45efc6ae-3d06-4f26-bb74-af2c6f8e7e97\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"
Jul 09 17:40:01.143048 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.143028    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79pdm\" (UniqueName: \"kubernetes.io/projected/00d27200-da1b-4d8a-9e38-65676147b344-kube-api-access-79pdm\") pod \"service-ca-operator-7695fc6c8c-rlwgs\" (UID: \"00d27200-da1b-4d8a-9e38-65676147b344\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs"
Jul 09 17:40:01.152814 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.152796    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs"
Jul 09 17:40:01.172055 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.172038    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j"
Jul 09 17:40:01.224146 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.224108    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.224282 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.224170    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8slz\" (UniqueName: \"kubernetes.io/projected/4faab181-b8e3-4a28-bc8f-0c59f5e55edd-kube-api-access-n8slz\") pod \"volume-data-source-validator-78495597f6-lqj4g\" (UID: \"4faab181-b8e3-4a28-bc8f-0c59f5e55edd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lqj4g"
Jul 09 17:40:01.224282 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.224205    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:40:01.224282 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.224244    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9604ff70-0e20-445a-94a5-ff74c0c9579c-serving-cert\") pod \"console-operator-7866497494-mj9vb\" (UID: \"9604ff70-0e20-445a-94a5-ff74c0c9579c\") " pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:01.224282 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.224269    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fb00f50-6d74-4c8c-9a7e-310f46e12b0f-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-nmqt5\" (UID: \"5fb00f50-6d74-4c8c-9a7e-310f46e12b0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5"
Jul 09 17:40:01.224492 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.224292    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-stats-auth\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.224492 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.224320    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5fb00f50-6d74-4c8c-9a7e-310f46e12b0f-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-nmqt5\" (UID: \"5fb00f50-6d74-4c8c-9a7e-310f46e12b0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5"
Jul 09 17:40:01.224492 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.224342    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjd2t\" (UniqueName: \"kubernetes.io/projected/d6450393-0509-4bb5-b799-9c97d8ce5c9e-kube-api-access-qjd2t\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.224492 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.224403    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb2mm\" (UniqueName: \"kubernetes.io/projected/9604ff70-0e20-445a-94a5-ff74c0c9579c-kube-api-access-xb2mm\") pod \"console-operator-7866497494-mj9vb\" (UID: \"9604ff70-0e20-445a-94a5-ff74c0c9579c\") " pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:01.224492 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.224431    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nr4t\" (UniqueName: \"kubernetes.io/projected/5fb00f50-6d74-4c8c-9a7e-310f46e12b0f-kube-api-access-7nr4t\") pod \"kube-storage-version-migrator-operator-799cb7f958-nmqt5\" (UID: \"5fb00f50-6d74-4c8c-9a7e-310f46e12b0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5"
Jul 09 17:40:01.224492 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.224467    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:40:01.224492 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.224491    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9604ff70-0e20-445a-94a5-ff74c0c9579c-config\") pod \"console-operator-7866497494-mj9vb\" (UID: \"9604ff70-0e20-445a-94a5-ff74c0c9579c\") " pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:01.224837 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.224516    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-default-certificate\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.224837 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.224593    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:40:01.224837 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.224645    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls podName:c71f2394-9aa0-40bf-a07c-26cd48a93a1c nodeName:}" failed. No retries permitted until 2026-07-09 17:40:17.224628391 +0000 UTC m=+61.238157484 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls") pod "dns-default-675dp" (UID: "c71f2394-9aa0-40bf-a07c-26cd48a93a1c") : secret "dns-default-metrics-tls" not found
Jul 09 17:40:01.224837 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.224597    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 17:40:01.224837 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.224660    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558c646897-dqv7k: secret "image-registry-tls" not found
Jul 09 17:40:01.224837 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.224664    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.224837 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.224693    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls podName:2e765850-1ceb-4508-a2d8-876bf7fd3a69 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:17.224682173 +0000 UTC m=+61.238211274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls") pod "image-registry-558c646897-dqv7k" (UID: "2e765850-1ceb-4508-a2d8-876bf7fd3a69") : secret "image-registry-tls" not found
Jul 09 17:40:01.224837 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.224751    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9604ff70-0e20-445a-94a5-ff74c0c9579c-trusted-ca\") pod \"console-operator-7866497494-mj9vb\" (UID: \"9604ff70-0e20-445a-94a5-ff74c0c9579c\") " pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:01.239264 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.239179    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8slz\" (UniqueName: \"kubernetes.io/projected/4faab181-b8e3-4a28-bc8f-0c59f5e55edd-kube-api-access-n8slz\") pod \"volume-data-source-validator-78495597f6-lqj4g\" (UID: \"4faab181-b8e3-4a28-bc8f-0c59f5e55edd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lqj4g"
Jul 09 17:40:01.249511 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.249407    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lqj4g"
Jul 09 17:40:01.311135 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:40:01.311095    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d0d87c_7e14_4ad2_8e16_b125198c2820.slice/crio-1877ae67b3c0f7bdfe7603b90ed7e9c328351009c410d54e8a1f47191d54ba04 WatchSource:0}: Error finding container 1877ae67b3c0f7bdfe7603b90ed7e9c328351009c410d54e8a1f47191d54ba04: Status 404 returned error can't find the container with id 1877ae67b3c0f7bdfe7603b90ed7e9c328351009c410d54e8a1f47191d54ba04
Jul 09 17:40:01.312271 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.312202    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-dkxlh"]
Jul 09 17:40:01.325409 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.325380    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.325539 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.325437    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9604ff70-0e20-445a-94a5-ff74c0c9579c-serving-cert\") pod \"console-operator-7866497494-mj9vb\" (UID: \"9604ff70-0e20-445a-94a5-ff74c0c9579c\") " pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:01.325539 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.325475    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fb00f50-6d74-4c8c-9a7e-310f46e12b0f-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-nmqt5\" (UID: \"5fb00f50-6d74-4c8c-9a7e-310f46e12b0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5"
Jul 09 17:40:01.325539 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.325495    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-stats-auth\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.325539 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.325520    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5fb00f50-6d74-4c8c-9a7e-310f46e12b0f-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-nmqt5\" (UID: \"5fb00f50-6d74-4c8c-9a7e-310f46e12b0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5"
Jul 09 17:40:01.325746 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.325546    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 17:40:01.325746 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.325615    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs podName:d6450393-0509-4bb5-b799-9c97d8ce5c9e nodeName:}" failed. No retries permitted until 2026-07-09 17:40:01.825595409 +0000 UTC m=+45.839124514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs") pod "router-default-d6b67c788-g98z2" (UID: "d6450393-0509-4bb5-b799-9c97d8ce5c9e") : secret "router-metrics-certs-default" not found
Jul 09 17:40:01.326768 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.326430    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5fb00f50-6d74-4c8c-9a7e-310f46e12b0f-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-nmqt5\" (UID: \"5fb00f50-6d74-4c8c-9a7e-310f46e12b0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5"
Jul 09 17:40:01.326768 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.325552    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjd2t\" (UniqueName: \"kubernetes.io/projected/d6450393-0509-4bb5-b799-9c97d8ce5c9e-kube-api-access-qjd2t\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.326768 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.326564    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb2mm\" (UniqueName: \"kubernetes.io/projected/9604ff70-0e20-445a-94a5-ff74c0c9579c-kube-api-access-xb2mm\") pod \"console-operator-7866497494-mj9vb\" (UID: \"9604ff70-0e20-445a-94a5-ff74c0c9579c\") " pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:01.327350 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.327190    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs"]
Jul 09 17:40:01.327640 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.327529    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nr4t\" (UniqueName: \"kubernetes.io/projected/5fb00f50-6d74-4c8c-9a7e-310f46e12b0f-kube-api-access-7nr4t\") pod \"kube-storage-version-migrator-operator-799cb7f958-nmqt5\" (UID: \"5fb00f50-6d74-4c8c-9a7e-310f46e12b0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5"
Jul 09 17:40:01.327938 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.327801    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9604ff70-0e20-445a-94a5-ff74c0c9579c-config\") pod \"console-operator-7866497494-mj9vb\" (UID: \"9604ff70-0e20-445a-94a5-ff74c0c9579c\") " pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:01.327938 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.327848    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-default-certificate\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.327938 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.327875    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.327938 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.327914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9604ff70-0e20-445a-94a5-ff74c0c9579c-trusted-ca\") pod \"console-operator-7866497494-mj9vb\" (UID: \"9604ff70-0e20-445a-94a5-ff74c0c9579c\") " pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:01.329170 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.328608    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle podName:d6450393-0509-4bb5-b799-9c97d8ce5c9e nodeName:}" failed. No retries permitted until 2026-07-09 17:40:01.828585125 +0000 UTC m=+45.842114237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle") pod "router-default-d6b67c788-g98z2" (UID: "d6450393-0509-4bb5-b799-9c97d8ce5c9e") : configmap references non-existent config key: service-ca.crt
Jul 09 17:40:01.329170 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.329140    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9604ff70-0e20-445a-94a5-ff74c0c9579c-config\") pod \"console-operator-7866497494-mj9vb\" (UID: \"9604ff70-0e20-445a-94a5-ff74c0c9579c\") " pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:01.329823 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.329504    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9604ff70-0e20-445a-94a5-ff74c0c9579c-trusted-ca\") pod \"console-operator-7866497494-mj9vb\" (UID: \"9604ff70-0e20-445a-94a5-ff74c0c9579c\") " pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:01.329823 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.329543    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9604ff70-0e20-445a-94a5-ff74c0c9579c-serving-cert\") pod \"console-operator-7866497494-mj9vb\" (UID: \"9604ff70-0e20-445a-94a5-ff74c0c9579c\") " pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:01.329823 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.329586    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-stats-auth\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.329823 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.329713    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fb00f50-6d74-4c8c-9a7e-310f46e12b0f-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-nmqt5\" (UID: \"5fb00f50-6d74-4c8c-9a7e-310f46e12b0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5"
Jul 09 17:40:01.330186 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:40:01.330159    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d27200_da1b_4d8a_9e38_65676147b344.slice/crio-35d695b9e26d01f8fd2f2f88c10b686a426ba5d352fd077ed06d80a3c26bbda7 WatchSource:0}: Error finding container 35d695b9e26d01f8fd2f2f88c10b686a426ba5d352fd077ed06d80a3c26bbda7: Status 404 returned error can't find the container with id 35d695b9e26d01f8fd2f2f88c10b686a426ba5d352fd077ed06d80a3c26bbda7
Jul 09 17:40:01.331110 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.331090    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-default-certificate\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.341327 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.340864    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjd2t\" (UniqueName: \"kubernetes.io/projected/d6450393-0509-4bb5-b799-9c97d8ce5c9e-kube-api-access-qjd2t\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.342797 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.342772    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb2mm\" (UniqueName: \"kubernetes.io/projected/9604ff70-0e20-445a-94a5-ff74c0c9579c-kube-api-access-xb2mm\") pod \"console-operator-7866497494-mj9vb\" (UID: \"9604ff70-0e20-445a-94a5-ff74c0c9579c\") " pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:01.343937 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.343910    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nr4t\" (UniqueName: \"kubernetes.io/projected/5fb00f50-6d74-4c8c-9a7e-310f46e12b0f-kube-api-access-7nr4t\") pod \"kube-storage-version-migrator-operator-799cb7f958-nmqt5\" (UID: \"5fb00f50-6d74-4c8c-9a7e-310f46e12b0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5"
Jul 09 17:40:01.354370 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.354344    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-xkn6j"]
Jul 09 17:40:01.356120 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.356103    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:01.357901 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:40:01.357856    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2bee9d_5d0f_4c62_a69b_b8e7993d858a.slice/crio-1c26dd815346a18ab72a6a4c389d3e6fb42faf67e831a0bd506ec62242651828 WatchSource:0}: Error finding container 1c26dd815346a18ab72a6a4c389d3e6fb42faf67e831a0bd506ec62242651828: Status 404 returned error can't find the container with id 1c26dd815346a18ab72a6a4c389d3e6fb42faf67e831a0bd506ec62242651828
Jul 09 17:40:01.425298 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.425271    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lqj4g"]
Jul 09 17:40:01.427151 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:40:01.427123    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4faab181_b8e3_4a28_bc8f_0c59f5e55edd.slice/crio-005340dd3e57efd270389284670f1813d5e96757c8546b8cdd350a76337ccd22 WatchSource:0}: Error finding container 005340dd3e57efd270389284670f1813d5e96757c8546b8cdd350a76337ccd22: Status 404 returned error can't find the container with id 005340dd3e57efd270389284670f1813d5e96757c8546b8cdd350a76337ccd22
Jul 09 17:40:01.428421 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.428382    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wp6zp\" (UID: \"de82e36c-3a74-425b-b950-6983d4686d0a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:01.428700 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.428602    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 17:40:01.428700 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.428680    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls podName:de82e36c-3a74-425b-b950-6983d4686d0a nodeName:}" failed. No retries permitted until 2026-07-09 17:40:02.428657548 +0000 UTC m=+46.442186694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-wp6zp" (UID: "de82e36c-3a74-425b-b950-6983d4686d0a") : secret "cluster-monitoring-operator-tls" not found
Jul 09 17:40:01.515673 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.515626    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-mj9vb"]
Jul 09 17:40:01.519651 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:40:01.519628    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9604ff70_0e20_445a_94a5_ff74c0c9579c.slice/crio-438cfdfdea4b8eb1f5b2ac7439e4504c2cc2ac7201baefc43966e904640b2789 WatchSource:0}: Error finding container 438cfdfdea4b8eb1f5b2ac7439e4504c2cc2ac7201baefc43966e904640b2789: Status 404 returned error can't find the container with id 438cfdfdea4b8eb1f5b2ac7439e4504c2cc2ac7201baefc43966e904640b2789
Jul 09 17:40:01.629841 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.629817    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wgnvc\" (UID: \"45efc6ae-3d06-4f26-bb74-af2c6f8e7e97\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"
Jul 09 17:40:01.629948 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.629939    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 17:40:01.629993 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.629987    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls podName:45efc6ae-3d06-4f26-bb74-af2c6f8e7e97 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:02.629973322 +0000 UTC m=+46.643502414 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls") pod "cluster-samples-operator-65db858894-wgnvc" (UID: "45efc6ae-3d06-4f26-bb74-af2c6f8e7e97") : secret "samples-operator-tls" not found
Jul 09 17:40:01.639133 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.639112    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5"
Jul 09 17:40:01.782258 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.782236    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5"]
Jul 09 17:40:01.784275 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:40:01.784238    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb00f50_6d74_4c8c_9a7e_310f46e12b0f.slice/crio-dbe23d6bfc2a0a2f141dfc09e8d229355227e08dd9f7d097b69bc40d462752e2 WatchSource:0}: Error finding container dbe23d6bfc2a0a2f141dfc09e8d229355227e08dd9f7d097b69bc40d462752e2: Status 404 returned error can't find the container with id dbe23d6bfc2a0a2f141dfc09e8d229355227e08dd9f7d097b69bc40d462752e2
Jul 09 17:40:01.831941 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.831909    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.832057 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.832039    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:01.832129 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.832060    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 17:40:01.832129 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.832114    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs podName:d6450393-0509-4bb5-b799-9c97d8ce5c9e nodeName:}" failed. No retries permitted until 2026-07-09 17:40:02.832100446 +0000 UTC m=+46.845629540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs") pod "router-default-d6b67c788-g98z2" (UID: "d6450393-0509-4bb5-b799-9c97d8ce5c9e") : secret "router-metrics-certs-default" not found
Jul 09 17:40:01.832232 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:01.832178    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle podName:d6450393-0509-4bb5-b799-9c97d8ce5c9e nodeName:}" failed. No retries permitted until 2026-07-09 17:40:02.832160506 +0000 UTC m=+46.845689616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle") pod "router-default-d6b67c788-g98z2" (UID: "d6450393-0509-4bb5-b799-9c97d8ce5c9e") : configmap references non-existent config key: service-ca.crt
Jul 09 17:40:01.847417 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.847385    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lqj4g" event={"ID":"4faab181-b8e3-4a28-bc8f-0c59f5e55edd","Type":"ContainerStarted","Data":"005340dd3e57efd270389284670f1813d5e96757c8546b8cdd350a76337ccd22"}
Jul 09 17:40:01.848520 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.848500    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j" event={"ID":"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a","Type":"ContainerStarted","Data":"1c26dd815346a18ab72a6a4c389d3e6fb42faf67e831a0bd506ec62242651828"}
Jul 09 17:40:01.849594 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.849572    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs" event={"ID":"00d27200-da1b-4d8a-9e38-65676147b344","Type":"ContainerStarted","Data":"35d695b9e26d01f8fd2f2f88c10b686a426ba5d352fd077ed06d80a3c26bbda7"}
Jul 09 17:40:01.850682 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.850660    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5" event={"ID":"5fb00f50-6d74-4c8c-9a7e-310f46e12b0f","Type":"ContainerStarted","Data":"dbe23d6bfc2a0a2f141dfc09e8d229355227e08dd9f7d097b69bc40d462752e2"}
Jul 09 17:40:01.852277 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.852194    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dkxlh" event={"ID":"04d0d87c-7e14-4ad2-8e16-b125198c2820","Type":"ContainerStarted","Data":"7ae0c94c3b719e4cef4be2b6a7a6e2d5323863f55c0779d346ae38a0d768c9ed"}
Jul 09 17:40:01.852277 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.852262    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dkxlh" event={"ID":"04d0d87c-7e14-4ad2-8e16-b125198c2820","Type":"ContainerStarted","Data":"1877ae67b3c0f7bdfe7603b90ed7e9c328351009c410d54e8a1f47191d54ba04"}
Jul 09 17:40:01.853218 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.853198    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mj9vb" event={"ID":"9604ff70-0e20-445a-94a5-ff74c0c9579c","Type":"ContainerStarted","Data":"438cfdfdea4b8eb1f5b2ac7439e4504c2cc2ac7201baefc43966e904640b2789"}
Jul 09 17:40:01.870125 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:01.870088    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dkxlh" podStartSLOduration=1.8700774930000001 podStartE2EDuration="1.870077493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:01.869847202 +0000 UTC m=+45.883376316" watchObservedRunningTime="2026-07-09 17:40:01.870077493 +0000 UTC m=+45.883606606"
Jul 09 17:40:02.437103 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:02.437017    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wp6zp\" (UID: \"de82e36c-3a74-425b-b950-6983d4686d0a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:02.437280 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:02.437190    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 17:40:02.437280 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:02.437266    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls podName:de82e36c-3a74-425b-b950-6983d4686d0a nodeName:}" failed. No retries permitted until 2026-07-09 17:40:04.437245563 +0000 UTC m=+48.450774672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-wp6zp" (UID: "de82e36c-3a74-425b-b950-6983d4686d0a") : secret "cluster-monitoring-operator-tls" not found
Jul 09 17:40:02.638737 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:02.638679    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wgnvc\" (UID: \"45efc6ae-3d06-4f26-bb74-af2c6f8e7e97\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"
Jul 09 17:40:02.638954 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:02.638938    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 17:40:02.639023 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:02.639006    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls podName:45efc6ae-3d06-4f26-bb74-af2c6f8e7e97 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:04.638988229 +0000 UTC m=+48.652517324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls") pod "cluster-samples-operator-65db858894-wgnvc" (UID: "45efc6ae-3d06-4f26-bb74-af2c6f8e7e97") : secret "samples-operator-tls" not found
Jul 09 17:40:02.841131 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:02.841092    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:02.841577 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:02.841242    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:02.841577 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:02.841377    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle podName:d6450393-0509-4bb5-b799-9c97d8ce5c9e nodeName:}" failed. No retries permitted until 2026-07-09 17:40:04.841357434 +0000 UTC m=+48.854886527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle") pod "router-default-d6b67c788-g98z2" (UID: "d6450393-0509-4bb5-b799-9c97d8ce5c9e") : configmap references non-existent config key: service-ca.crt
Jul 09 17:40:02.841811 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:02.841793    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 17:40:02.841880 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:02.841852    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs podName:d6450393-0509-4bb5-b799-9c97d8ce5c9e nodeName:}" failed. No retries permitted until 2026-07-09 17:40:04.84183556 +0000 UTC m=+48.855364658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs") pod "router-default-d6b67c788-g98z2" (UID: "d6450393-0509-4bb5-b799-9c97d8ce5c9e") : secret "router-metrics-certs-default" not found
Jul 09 17:40:04.457388 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:04.457341    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wp6zp\" (UID: \"de82e36c-3a74-425b-b950-6983d4686d0a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:04.457905 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:04.457531    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 17:40:04.457905 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:04.457604    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls podName:de82e36c-3a74-425b-b950-6983d4686d0a nodeName:}" failed. No retries permitted until 2026-07-09 17:40:08.457583631 +0000 UTC m=+52.471112739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-wp6zp" (UID: "de82e36c-3a74-425b-b950-6983d4686d0a") : secret "cluster-monitoring-operator-tls" not found
Jul 09 17:40:04.659277 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:04.659237    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wgnvc\" (UID: \"45efc6ae-3d06-4f26-bb74-af2c6f8e7e97\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"
Jul 09 17:40:04.659450 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:04.659405    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 17:40:04.659515 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:04.659478    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls podName:45efc6ae-3d06-4f26-bb74-af2c6f8e7e97 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:08.659461856 +0000 UTC m=+52.672990948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls") pod "cluster-samples-operator-65db858894-wgnvc" (UID: "45efc6ae-3d06-4f26-bb74-af2c6f8e7e97") : secret "samples-operator-tls" not found
Jul 09 17:40:04.861641 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:04.861602    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:04.861853 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:04.861678    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:04.861853 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:04.861812    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle podName:d6450393-0509-4bb5-b799-9c97d8ce5c9e nodeName:}" failed. No retries permitted until 2026-07-09 17:40:08.861786368 +0000 UTC m=+52.875315474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle") pod "router-default-d6b67c788-g98z2" (UID: "d6450393-0509-4bb5-b799-9c97d8ce5c9e") : configmap references non-existent config key: service-ca.crt
Jul 09 17:40:04.861853 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:04.861844    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 17:40:04.862021 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:04.861906    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs podName:d6450393-0509-4bb5-b799-9c97d8ce5c9e nodeName:}" failed. No retries permitted until 2026-07-09 17:40:08.861890294 +0000 UTC m=+52.875419390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs") pod "router-default-d6b67c788-g98z2" (UID: "d6450393-0509-4bb5-b799-9c97d8ce5c9e") : secret "router-metrics-certs-default" not found
Jul 09 17:40:05.870244 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:05.870176    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/0.log"
Jul 09 17:40:05.870244 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:05.870213    2573 generic.go:356] "Generic (PLEG): container finished" podID="9604ff70-0e20-445a-94a5-ff74c0c9579c" containerID="6faf23973cd0bd509f109ed25d79ff6c2588665b44ecfd1d4f38e0d95f800c3e" exitCode=255
Jul 09 17:40:05.870685 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:05.870300    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mj9vb" event={"ID":"9604ff70-0e20-445a-94a5-ff74c0c9579c","Type":"ContainerDied","Data":"6faf23973cd0bd509f109ed25d79ff6c2588665b44ecfd1d4f38e0d95f800c3e"}
Jul 09 17:40:05.870685 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:05.870520    2573 scope.go:117] "RemoveContainer" containerID="6faf23973cd0bd509f109ed25d79ff6c2588665b44ecfd1d4f38e0d95f800c3e"
Jul 09 17:40:05.872084 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:05.872054    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lqj4g" event={"ID":"4faab181-b8e3-4a28-bc8f-0c59f5e55edd","Type":"ContainerStarted","Data":"3e35ff41641836472dc66a4ac6d8ee8d274a943c9c8bdec9b3bafe1c94938db5"}
Jul 09 17:40:05.874093 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:05.874048    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j" event={"ID":"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a","Type":"ContainerStarted","Data":"7e0de5ca70aa5fbc7121e86c7bef1e0518cb39f16f5fd5b2e3c6344e96b16c4e"}
Jul 09 17:40:05.876382 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:05.876216    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs" event={"ID":"00d27200-da1b-4d8a-9e38-65676147b344","Type":"ContainerStarted","Data":"e95b26ef568100999488881086dfe5171e9da7bbe26963e8c37275b510d1a389"}
Jul 09 17:40:05.877949 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:05.877925    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5" event={"ID":"5fb00f50-6d74-4c8c-9a7e-310f46e12b0f","Type":"ContainerStarted","Data":"673eef9cae209c5f632dfad034fe842759d96b0b283447616f3bb4644ed2c217"}
Jul 09 17:40:05.908456 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:05.908409    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lqj4g" podStartSLOduration=1.789766702 podStartE2EDuration="5.908392912s" totalImagesPullingTime="4.11862621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:05.907630988 +0000 UTC m=+49.921160103" watchObservedRunningTime="2026-07-09 17:40:05.908392912 +0000 UTC m=+49.921922023"
Jul 09 17:40:05.929216 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:05.929170    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs" podStartSLOduration=1.712954142 podStartE2EDuration="5.92915301s" totalImagesPullingTime="4.216198868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:05.927641584 +0000 UTC m=+49.941170699" watchObservedRunningTime="2026-07-09 17:40:05.92915301 +0000 UTC m=+49.942682125"
Jul 09 17:40:05.970370 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:05.970315    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j" podStartSLOduration=1.776462368 podStartE2EDuration="5.970297584s" totalImagesPullingTime="4.193835216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:05.949655097 +0000 UTC m=+49.963184213" watchObservedRunningTime="2026-07-09 17:40:05.970297584 +0000 UTC m=+49.983826702"
Jul 09 17:40:05.971670 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:05.971626    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5" podStartSLOduration=2.209364663 podStartE2EDuration="5.971614089s" totalImagesPullingTime="3.762249426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:05.969262614 +0000 UTC m=+49.982791727" watchObservedRunningTime="2026-07-09 17:40:05.971614089 +0000 UTC m=+49.985143203"
Jul 09 17:40:06.883052 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:06.882984    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/1.log"
Jul 09 17:40:06.883425 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:06.883351    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/0.log"
Jul 09 17:40:06.883425 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:06.883381    2573 generic.go:356] "Generic (PLEG): container finished" podID="9604ff70-0e20-445a-94a5-ff74c0c9579c" containerID="a4430972efc4eb95cee70b4a10c01196fb8c802f9840de630cb0148991dbe70a" exitCode=255
Jul 09 17:40:06.883522 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:06.883496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mj9vb" event={"ID":"9604ff70-0e20-445a-94a5-ff74c0c9579c","Type":"ContainerDied","Data":"a4430972efc4eb95cee70b4a10c01196fb8c802f9840de630cb0148991dbe70a"}
Jul 09 17:40:06.883578 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:06.883566    2573 scope.go:117] "RemoveContainer" containerID="6faf23973cd0bd509f109ed25d79ff6c2588665b44ecfd1d4f38e0d95f800c3e"
Jul 09 17:40:06.883824 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:06.883801    2573 scope.go:117] "RemoveContainer" containerID="a4430972efc4eb95cee70b4a10c01196fb8c802f9840de630cb0148991dbe70a"
Jul 09 17:40:06.884049 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:06.884026    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-mj9vb_openshift-console-operator(9604ff70-0e20-445a-94a5-ff74c0c9579c)\"" pod="openshift-console-operator/console-operator-7866497494-mj9vb" podUID="9604ff70-0e20-445a-94a5-ff74c0c9579c"
Jul 09 17:40:07.889065 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:07.889035    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/1.log"
Jul 09 17:40:07.889589 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:07.889445    2573 scope.go:117] "RemoveContainer" containerID="a4430972efc4eb95cee70b4a10c01196fb8c802f9840de630cb0148991dbe70a"
Jul 09 17:40:07.889655 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:07.889624    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-mj9vb_openshift-console-operator(9604ff70-0e20-445a-94a5-ff74c0c9579c)\"" pod="openshift-console-operator/console-operator-7866497494-mj9vb" podUID="9604ff70-0e20-445a-94a5-ff74c0c9579c"
Jul 09 17:40:08.003748 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.003700    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-54d9n"]
Jul 09 17:40:08.008885 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.008869    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.028923 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.028897    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 17:40:08.029026 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.028897    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 17:40:08.029530 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.029513    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4s5jj\""
Jul 09 17:40:08.056622 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.056594    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-54d9n"]
Jul 09 17:40:08.089619 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.089588    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81dade0c-92b2-44a3-9b6d-5ba9265259a3-data-volume\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.089754 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.089635    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.089754 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.089654    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnv2v\" (UniqueName: \"kubernetes.io/projected/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-api-access-bnv2v\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.089859 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.089785    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.089859 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.089811    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81dade0c-92b2-44a3-9b6d-5ba9265259a3-crio-socket\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.190500 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.190416    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81dade0c-92b2-44a3-9b6d-5ba9265259a3-data-volume\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.190500 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.190478    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.190677 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.190503    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnv2v\" (UniqueName: \"kubernetes.io/projected/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-api-access-bnv2v\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.190677 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.190578    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.190677 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.190601    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81dade0c-92b2-44a3-9b6d-5ba9265259a3-crio-socket\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.190677 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:08.190626    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:08.190949 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:08.190676    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 17:40:08.190949 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:08.190690    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls podName:81dade0c-92b2-44a3-9b6d-5ba9265259a3 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:08.690675118 +0000 UTC m=+52.704204214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54d9n" (UID: "81dade0c-92b2-44a3-9b6d-5ba9265259a3") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:08.190949 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.190805    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81dade0c-92b2-44a3-9b6d-5ba9265259a3-crio-socket\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.190949 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.190837    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81dade0c-92b2-44a3-9b6d-5ba9265259a3-data-volume\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.190949 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:08.190850    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm podName:81dade0c-92b2-44a3-9b6d-5ba9265259a3 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:08.690831811 +0000 UTC m=+52.704360903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-54d9n" (UID: "81dade0c-92b2-44a3-9b6d-5ba9265259a3") : configmap "kube-rbac-proxy" not found
Jul 09 17:40:08.210009 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.209988    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnv2v\" (UniqueName: \"kubernetes.io/projected/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-api-access-bnv2v\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.493995 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.493960    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wp6zp\" (UID: \"de82e36c-3a74-425b-b950-6983d4686d0a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:08.494151 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:08.494094    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 17:40:08.494151 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:08.494145    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls podName:de82e36c-3a74-425b-b950-6983d4686d0a nodeName:}" failed. No retries permitted until 2026-07-09 17:40:16.4941328 +0000 UTC m=+60.507661892 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-wp6zp" (UID: "de82e36c-3a74-425b-b950-6983d4686d0a") : secret "cluster-monitoring-operator-tls" not found
Jul 09 17:40:08.695294 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.695259    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.695476 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.695334    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:08.695476 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.695363    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wgnvc\" (UID: \"45efc6ae-3d06-4f26-bb74-af2c6f8e7e97\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"
Jul 09 17:40:08.695476 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:08.695407    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:08.695476 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:08.695445    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 17:40:08.695476 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:08.695464    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 17:40:08.695476 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:08.695480    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls podName:81dade0c-92b2-44a3-9b6d-5ba9265259a3 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:09.695465313 +0000 UTC m=+53.708994405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54d9n" (UID: "81dade0c-92b2-44a3-9b6d-5ba9265259a3") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:08.695688 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:08.695495    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm podName:81dade0c-92b2-44a3-9b6d-5ba9265259a3 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:09.695488826 +0000 UTC m=+53.709017918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-54d9n" (UID: "81dade0c-92b2-44a3-9b6d-5ba9265259a3") : configmap "kube-rbac-proxy" not found
Jul 09 17:40:08.695688 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:08.695505    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls podName:45efc6ae-3d06-4f26-bb74-af2c6f8e7e97 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:16.695500347 +0000 UTC m=+60.709029438 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls") pod "cluster-samples-operator-65db858894-wgnvc" (UID: "45efc6ae-3d06-4f26-bb74-af2c6f8e7e97") : secret "samples-operator-tls" not found
Jul 09 17:40:08.897126 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.897038    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:08.897126 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:08.897088    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:08.897502 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:08.897196    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 17:40:08.897502 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:08.897212    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle podName:d6450393-0509-4bb5-b799-9c97d8ce5c9e nodeName:}" failed. No retries permitted until 2026-07-09 17:40:16.897194028 +0000 UTC m=+60.910723137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle") pod "router-default-d6b67c788-g98z2" (UID: "d6450393-0509-4bb5-b799-9c97d8ce5c9e") : configmap references non-existent config key: service-ca.crt
Jul 09 17:40:08.897502 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:08.897233    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs podName:d6450393-0509-4bb5-b799-9c97d8ce5c9e nodeName:}" failed. No retries permitted until 2026-07-09 17:40:16.897223306 +0000 UTC m=+60.910752407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs") pod "router-default-d6b67c788-g98z2" (UID: "d6450393-0509-4bb5-b799-9c97d8ce5c9e") : secret "router-metrics-certs-default" not found
Jul 09 17:40:09.703211 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:09.703132    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:09.703211 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:09.703195    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 17:40:09.703515 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:09.703266    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm podName:81dade0c-92b2-44a3-9b6d-5ba9265259a3 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:11.703253032 +0000 UTC m=+55.716782124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-54d9n" (UID: "81dade0c-92b2-44a3-9b6d-5ba9265259a3") : configmap "kube-rbac-proxy" not found
Jul 09 17:40:09.703515 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:09.703339    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:09.703515 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:09.703425    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:09.703515 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:09.703453    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls podName:81dade0c-92b2-44a3-9b6d-5ba9265259a3 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:11.703445674 +0000 UTC m=+55.716974767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54d9n" (UID: "81dade0c-92b2-44a3-9b6d-5ba9265259a3") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:10.559923 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:10.559895    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cctw4_a3dcc235-2a55-4e67-a42c-4d047739f8f0/dns-node-resolver/0.log"
Jul 09 17:40:11.356880 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:11.356846    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:11.357044 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:11.356899    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:11.357226 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:11.357214    2573 scope.go:117] "RemoveContainer" containerID="a4430972efc4eb95cee70b4a10c01196fb8c802f9840de630cb0148991dbe70a"
Jul 09 17:40:11.357372 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:11.357355    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-mj9vb_openshift-console-operator(9604ff70-0e20-445a-94a5-ff74c0c9579c)\"" pod="openshift-console-operator/console-operator-7866497494-mj9vb" podUID="9604ff70-0e20-445a-94a5-ff74c0c9579c"
Jul 09 17:40:11.718924 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:11.718836    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:11.718924 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:11.718919    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:11.719363 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:11.718994    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:11.719363 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:11.719051    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls podName:81dade0c-92b2-44a3-9b6d-5ba9265259a3 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:15.719037276 +0000 UTC m=+59.732566368 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54d9n" (UID: "81dade0c-92b2-44a3-9b6d-5ba9265259a3") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:11.719363 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:11.719054    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 17:40:11.719363 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:11.719120    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm podName:81dade0c-92b2-44a3-9b6d-5ba9265259a3 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:15.719103513 +0000 UTC m=+59.732632608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-54d9n" (UID: "81dade0c-92b2-44a3-9b6d-5ba9265259a3") : configmap "kube-rbac-proxy" not found
Jul 09 17:40:11.761890 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:11.761867    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jbqs8_76f015b9-8aaa-40ee-a73e-492fb91418d0/node-ca/0.log"
Jul 09 17:40:12.559871 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:12.559840    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9rw5k_00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517/serve-healthcheck-canary/0.log"
Jul 09 17:40:13.364440 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:13.364412    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-nmqt5_5fb00f50-6d74-4c8c-9a7e-310f46e12b0f/kube-storage-version-migrator-operator/0.log"
Jul 09 17:40:15.755430 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:15.755397    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:15.755834 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:15.755458    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:15.755834 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:15.755534    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 17:40:15.755834 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:15.755583    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm podName:81dade0c-92b2-44a3-9b6d-5ba9265259a3 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:23.755569834 +0000 UTC m=+67.769098926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-54d9n" (UID: "81dade0c-92b2-44a3-9b6d-5ba9265259a3") : configmap "kube-rbac-proxy" not found
Jul 09 17:40:15.755834 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:15.755533    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:15.755834 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:15.755662    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls podName:81dade0c-92b2-44a3-9b6d-5ba9265259a3 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:23.755649613 +0000 UTC m=+67.769178705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54d9n" (UID: "81dade0c-92b2-44a3-9b6d-5ba9265259a3") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:16.563632 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:16.563596    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wp6zp\" (UID: \"de82e36c-3a74-425b-b950-6983d4686d0a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:16.563815 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:16.563751    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 17:40:16.563815 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:16.563810    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls podName:de82e36c-3a74-425b-b950-6983d4686d0a nodeName:}" failed. No retries permitted until 2026-07-09 17:40:32.563793627 +0000 UTC m=+76.577322722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-wp6zp" (UID: "de82e36c-3a74-425b-b950-6983d4686d0a") : secret "cluster-monitoring-operator-tls" not found
Jul 09 17:40:16.765428 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:16.765386    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wgnvc\" (UID: \"45efc6ae-3d06-4f26-bb74-af2c6f8e7e97\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"
Jul 09 17:40:16.767667 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:16.767642    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45efc6ae-3d06-4f26-bb74-af2c6f8e7e97-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wgnvc\" (UID: \"45efc6ae-3d06-4f26-bb74-af2c6f8e7e97\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"
Jul 09 17:40:16.816529 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:16.816465    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cd8xx\""
Jul 09 17:40:16.824224 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:16.824209    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"
Jul 09 17:40:16.967409 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:16.967379    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:16.967537 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:16.967424    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:16.967537 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:16.967511    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 17:40:16.967616 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:16.967539    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle podName:d6450393-0509-4bb5-b799-9c97d8ce5c9e nodeName:}" failed. No retries permitted until 2026-07-09 17:40:32.967522332 +0000 UTC m=+76.981051426 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle") pod "router-default-d6b67c788-g98z2" (UID: "d6450393-0509-4bb5-b799-9c97d8ce5c9e") : configmap references non-existent config key: service-ca.crt
Jul 09 17:40:16.967616 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:16.967605    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs podName:d6450393-0509-4bb5-b799-9c97d8ce5c9e nodeName:}" failed. No retries permitted until 2026-07-09 17:40:32.967586086 +0000 UTC m=+76.981115182 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs") pod "router-default-d6b67c788-g98z2" (UID: "d6450393-0509-4bb5-b799-9c97d8ce5c9e") : secret "router-metrics-certs-default" not found
Jul 09 17:40:16.982417 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:16.982396    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc"]
Jul 09 17:40:17.269857 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.269821    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:40:17.270056 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.269925    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:40:17.272269 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.272246    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls\") pod \"image-registry-558c646897-dqv7k\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") " pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:40:17.272369 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.272252    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c71f2394-9aa0-40bf-a07c-26cd48a93a1c-metrics-tls\") pod \"dns-default-675dp\" (UID: \"c71f2394-9aa0-40bf-a07c-26cd48a93a1c\") " pod="openshift-dns/dns-default-675dp"
Jul 09 17:40:17.458239 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.458163    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-48n2x\""
Jul 09 17:40:17.465143 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.465125    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:40:17.472260 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.472239    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2k2gp\""
Jul 09 17:40:17.480088 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.480066    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-675dp"
Jul 09 17:40:17.622677 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.622648    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-558c646897-dqv7k"]
Jul 09 17:40:17.626305 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:40:17.626274    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e765850_1ceb_4508_a2d8_876bf7fd3a69.slice/crio-c7c3c98e22f4c9992d378a429bb41f2a657427a9e3898c2543ba9a5e290f4910 WatchSource:0}: Error finding container c7c3c98e22f4c9992d378a429bb41f2a657427a9e3898c2543ba9a5e290f4910: Status 404 returned error can't find the container with id c7c3c98e22f4c9992d378a429bb41f2a657427a9e3898c2543ba9a5e290f4910
Jul 09 17:40:17.641391 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.641370    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-675dp"]
Jul 09 17:40:17.644796 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:40:17.644775    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71f2394_9aa0_40bf_a07c_26cd48a93a1c.slice/crio-fa212ff7a7dc372e90411190949479c6228a805d84f6c6f3725a99ed62617035 WatchSource:0}: Error finding container fa212ff7a7dc372e90411190949479c6228a805d84f6c6f3725a99ed62617035: Status 404 returned error can't find the container with id fa212ff7a7dc372e90411190949479c6228a805d84f6c6f3725a99ed62617035
Jul 09 17:40:17.923875 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.923820    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558c646897-dqv7k" event={"ID":"2e765850-1ceb-4508-a2d8-876bf7fd3a69","Type":"ContainerStarted","Data":"b7ffb2ef8180d558912f1c0da914ccbbc52974dbd9fb9534cb2b9ebfabf1468f"}
Jul 09 17:40:17.924335 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.923890    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:40:17.924335 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.923913    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558c646897-dqv7k" event={"ID":"2e765850-1ceb-4508-a2d8-876bf7fd3a69","Type":"ContainerStarted","Data":"c7c3c98e22f4c9992d378a429bb41f2a657427a9e3898c2543ba9a5e290f4910"}
Jul 09 17:40:17.925173 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.925140    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc" event={"ID":"45efc6ae-3d06-4f26-bb74-af2c6f8e7e97","Type":"ContainerStarted","Data":"c0d080656462de51d64af74e70ebb7d3f4583c4fb860093ec6791baef3f534bb"}
Jul 09 17:40:17.926506 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.926464    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-675dp" event={"ID":"c71f2394-9aa0-40bf-a07c-26cd48a93a1c","Type":"ContainerStarted","Data":"fa212ff7a7dc372e90411190949479c6228a805d84f6c6f3725a99ed62617035"}
Jul 09 17:40:17.948770 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:17.948701    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-558c646897-dqv7k" podStartSLOduration=60.948685472 podStartE2EDuration="1m0.948685472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:17.946955428 +0000 UTC m=+61.960484560" watchObservedRunningTime="2026-07-09 17:40:17.948685472 +0000 UTC m=+61.962214586"
Jul 09 17:40:20.939136 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:20.939100    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc" event={"ID":"45efc6ae-3d06-4f26-bb74-af2c6f8e7e97","Type":"ContainerStarted","Data":"0ca2932d4707125e1c326b0c6cefd5627e9972a983848e636c9f3394614faffb"}
Jul 09 17:40:20.939641 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:20.939168    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc" event={"ID":"45efc6ae-3d06-4f26-bb74-af2c6f8e7e97","Type":"ContainerStarted","Data":"bfbac96e9cf3ffdd27f6bfdd54cf61be15896144e8400fc033269558b596b38e"}
Jul 09 17:40:20.940888 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:20.940860    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-675dp" event={"ID":"c71f2394-9aa0-40bf-a07c-26cd48a93a1c","Type":"ContainerStarted","Data":"aadf2fbcd65d2cd0b51aaacf12ba57cc2dfae1f4e8db46713432bd662d8614fe"}
Jul 09 17:40:20.940994 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:20.940913    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-675dp"
Jul 09 17:40:20.940994 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:20.940932    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-675dp" event={"ID":"c71f2394-9aa0-40bf-a07c-26cd48a93a1c","Type":"ContainerStarted","Data":"062bd9de7e5fde8657b4278c336ada11189f3f287d3b3c90219bca3ac6b385c1"}
Jul 09 17:40:20.983144 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:20.983091    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-675dp" podStartSLOduration=33.425682942 podStartE2EDuration="35.983072999s" totalImagesPullingTime="2.557390057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:20.98233033 +0000 UTC m=+64.995859445" watchObservedRunningTime="2026-07-09 17:40:20.983072999 +0000 UTC m=+64.996602113"
Jul 09 17:40:20.983962 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:20.983934    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wgnvc" podStartSLOduration=17.85061982 podStartE2EDuration="20.983925603s" totalImagesPullingTime="3.133305783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:20.960954725 +0000 UTC m=+64.974483838" watchObservedRunningTime="2026-07-09 17:40:20.983925603 +0000 UTC m=+64.997454716"
Jul 09 17:40:21.105531 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:21.105492    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs\") pod \"network-metrics-daemon-6qvsq\" (UID: \"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc\") " pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:40:21.105680 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:21.105607    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 17:40:21.105680 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:21.105662    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs podName:8a62897b-ea47-4b2f-8762-a4fc7b7a18dc nodeName:}" failed. No retries permitted until 2026-07-09 17:41:25.105647747 +0000 UTC m=+129.119176839 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs") pod "network-metrics-daemon-6qvsq" (UID: "8a62897b-ea47-4b2f-8762-a4fc7b7a18dc") : secret "metrics-daemon-secret" not found
Jul 09 17:40:22.523386 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:22.523358    2573 scope.go:117] "RemoveContainer" containerID="a4430972efc4eb95cee70b4a10c01196fb8c802f9840de630cb0148991dbe70a"
Jul 09 17:40:22.949750 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:22.949665    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/1.log"
Jul 09 17:40:22.949897 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:22.949749    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mj9vb" event={"ID":"9604ff70-0e20-445a-94a5-ff74c0c9579c","Type":"ContainerStarted","Data":"23c4f82a7d3c61df388d7743a396fc198903e9559e1cef39971783979b6e99c9"}
Jul 09 17:40:22.950075 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:22.950049    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:22.974766 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:22.974686    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-mj9vb" podStartSLOduration=18.945348676000002 podStartE2EDuration="22.97466539s" totalImagesPullingTime="4.029316714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:22.971053563 +0000 UTC m=+66.984582697" watchObservedRunningTime="2026-07-09 17:40:22.97466539 +0000 UTC m=+66.988194506"
Jul 09 17:40:23.099926 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:23.099892    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-mj9vb"
Jul 09 17:40:23.826811 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:23.826777    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:23.827191 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:23.826841    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:23.827191 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:23.826943    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:23.827191 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:23.827009    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls podName:81dade0c-92b2-44a3-9b6d-5ba9265259a3 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:39.826994751 +0000 UTC m=+83.840523843 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54d9n" (UID: "81dade0c-92b2-44a3-9b6d-5ba9265259a3") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:23.827303 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:23.827279    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81dade0c-92b2-44a3-9b6d-5ba9265259a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:27.840039 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:27.840011    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wtdtc"
Jul 09 17:40:30.946361 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:30.946329    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-675dp"
Jul 09 17:40:32.597082 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:32.597047    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wp6zp\" (UID: \"de82e36c-3a74-425b-b950-6983d4686d0a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:32.599418 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:32.599396    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/de82e36c-3a74-425b-b950-6983d4686d0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-wp6zp\" (UID: \"de82e36c-3a74-425b-b950-6983d4686d0a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:32.821537 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:32.821507    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hjrxj\""
Jul 09 17:40:32.827661 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:32.827641    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"
Jul 09 17:40:32.953780 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:32.953752    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp"]
Jul 09 17:40:32.956110 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:40:32.956083    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde82e36c_3a74_425b_b950_6983d4686d0a.slice/crio-5ecabcf2217bf577d96e03aeea3a36d010c5c4c33cbc39afb1ade4982434b8c8 WatchSource:0}: Error finding container 5ecabcf2217bf577d96e03aeea3a36d010c5c4c33cbc39afb1ade4982434b8c8: Status 404 returned error can't find the container with id 5ecabcf2217bf577d96e03aeea3a36d010c5c4c33cbc39afb1ade4982434b8c8
Jul 09 17:40:32.983120 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:32.983091    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp" event={"ID":"de82e36c-3a74-425b-b950-6983d4686d0a","Type":"ContainerStarted","Data":"5ecabcf2217bf577d96e03aeea3a36d010c5c4c33cbc39afb1ade4982434b8c8"}
Jul 09 17:40:33.001481 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:33.001452    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:33.001616 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:33.001500    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:33.002173 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:33.002152    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6450393-0509-4bb5-b799-9c97d8ce5c9e-service-ca-bundle\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:33.003783 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:33.003764    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6450393-0509-4bb5-b799-9c97d8ce5c9e-metrics-certs\") pod \"router-default-d6b67c788-g98z2\" (UID: \"d6450393-0509-4bb5-b799-9c97d8ce5c9e\") " pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:33.181396 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:33.181321    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-msg4q\""
Jul 09 17:40:33.188951 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:33.188934    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:33.323200 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:33.323156    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d6b67c788-g98z2"]
Jul 09 17:40:33.325220 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:40:33.325197    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6450393_0509_4bb5_b799_9c97d8ce5c9e.slice/crio-fdf4a4016c34dc4f53ba9117dbe12e3aefe925524251fc2da8e97f57693b811a WatchSource:0}: Error finding container fdf4a4016c34dc4f53ba9117dbe12e3aefe925524251fc2da8e97f57693b811a: Status 404 returned error can't find the container with id fdf4a4016c34dc4f53ba9117dbe12e3aefe925524251fc2da8e97f57693b811a
Jul 09 17:40:33.988875 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:33.988829    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d6b67c788-g98z2" event={"ID":"d6450393-0509-4bb5-b799-9c97d8ce5c9e","Type":"ContainerStarted","Data":"7919981a44648fd996eeff6f200fb22c6970e05a97ad5a299a2e998b3d4304eb"}
Jul 09 17:40:33.989243 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:33.988882    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d6b67c788-g98z2" event={"ID":"d6450393-0509-4bb5-b799-9c97d8ce5c9e","Type":"ContainerStarted","Data":"fdf4a4016c34dc4f53ba9117dbe12e3aefe925524251fc2da8e97f57693b811a"}
Jul 09 17:40:34.020390 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:34.020333    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-d6b67c788-g98z2" podStartSLOduration=34.020319432 podStartE2EDuration="34.020319432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:34.019030877 +0000 UTC m=+78.032560019" watchObservedRunningTime="2026-07-09 17:40:34.020319432 +0000 UTC m=+78.033848546"
Jul 09 17:40:34.189924 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:34.189887    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:34.192846 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:34.192823    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:34.995754 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:34.995697    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp" event={"ID":"de82e36c-3a74-425b-b950-6983d4686d0a","Type":"ContainerStarted","Data":"677b39d100e233bc478ae644156056d7b367a47beed9eb493ed2a98c61efe7e8"}
Jul 09 17:40:34.996207 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:34.995776    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:34.997527 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:34.997502    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-d6b67c788-g98z2"
Jul 09 17:40:35.025799 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:35.025759    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-wp6zp" podStartSLOduration=33.513219107 podStartE2EDuration="35.025744624s" totalImagesPullingTime="1.512525517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:35.024252529 +0000 UTC m=+79.037781664" watchObservedRunningTime="2026-07-09 17:40:35.025744624 +0000 UTC m=+79.039273738"
Jul 09 17:40:37.469998 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:37.469964    2573 patch_prober.go:28] interesting pod/image-registry-558c646897-dqv7k container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 17:40:37.470471 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:37.470014    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-558c646897-dqv7k" podUID="2e765850-1ceb-4508-a2d8-876bf7fd3a69" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:40:38.935708 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:38.935591    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:40:39.861661 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:39.861615    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:39.863825 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:39.863805    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81dade0c-92b2-44a3-9b6d-5ba9265259a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54d9n\" (UID: \"81dade0c-92b2-44a3-9b6d-5ba9265259a3\") " pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:40.121602 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:40.121525    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4s5jj\""
Jul 09 17:40:40.128726 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:40.128698    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-54d9n"
Jul 09 17:40:40.262794 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:40.262711    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-54d9n"]
Jul 09 17:40:40.265268 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:40:40.265236    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81dade0c_92b2_44a3_9b6d_5ba9265259a3.slice/crio-d4ffaaa6d4257b2aeccb0271c6bd2729e158c318e0f49398cf556e22edfb3905 WatchSource:0}: Error finding container d4ffaaa6d4257b2aeccb0271c6bd2729e158c318e0f49398cf556e22edfb3905: Status 404 returned error can't find the container with id d4ffaaa6d4257b2aeccb0271c6bd2729e158c318e0f49398cf556e22edfb3905
Jul 09 17:40:41.020760 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:41.020665    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54d9n" event={"ID":"81dade0c-92b2-44a3-9b6d-5ba9265259a3","Type":"ContainerStarted","Data":"d4ffaaa6d4257b2aeccb0271c6bd2729e158c318e0f49398cf556e22edfb3905"}
Jul 09 17:40:45.036181 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:45.036146    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54d9n" event={"ID":"81dade0c-92b2-44a3-9b6d-5ba9265259a3","Type":"ContainerStarted","Data":"37264306b6498583e6c89f47e37edea1cc4feb834762a2e569a258dc86cc1201"}
Jul 09 17:40:45.036534 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:45.036201    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54d9n" event={"ID":"81dade0c-92b2-44a3-9b6d-5ba9265259a3","Type":"ContainerStarted","Data":"213e3510a33b2d2ec826f6501bafe2c87c6943dee27ba1c980a834cc1e1e23e8"}
Jul 09 17:40:46.726172 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.726143    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ms8ps"]
Jul 09 17:40:46.734548 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.734520    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-dnq47"]
Jul 09 17:40:46.734708 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.734655    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:46.738686 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.738660    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vtjz5\""
Jul 09 17:40:46.738841 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.738660    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 17:40:46.739222 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.739190    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:46.740144 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.740074    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 17:40:46.740144 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.740127    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 17:40:46.740331 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.740186    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 17:40:46.742388 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.742190    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 09 17:40:46.742509 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.742496    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 09 17:40:46.742751 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.742733    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wjn7c\""
Jul 09 17:40:46.742986 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.742970    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 09 17:40:46.753121 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.753060    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-dnq47"]
Jul 09 17:40:46.921837 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.921794    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f8c82b7-be76-45ff-8440-eddb547acf3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:46.922028 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.921861    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f8c82b7-be76-45ff-8440-eddb547acf3b-sys\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:46.922028 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.921923    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87a6d039-68e6-4655-bf1f-9552c7d5b789-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:46.922028 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.921961    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87a6d039-68e6-4655-bf1f-9552c7d5b789-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:46.922028 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.921997    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87a6d039-68e6-4655-bf1f-9552c7d5b789-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:46.922250 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.922034    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f8c82b7-be76-45ff-8440-eddb547acf3b-root\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:46.922250 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.922078    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87a6d039-68e6-4655-bf1f-9552c7d5b789-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:46.922250 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.922104    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-285zb\" (UniqueName: \"kubernetes.io/projected/87a6d039-68e6-4655-bf1f-9552c7d5b789-kube-api-access-285zb\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:46.922250 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.922167    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f8c82b7-be76-45ff-8440-eddb547acf3b-node-exporter-textfile\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:46.922250 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.922223    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f8c82b7-be76-45ff-8440-eddb547acf3b-metrics-client-ca\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:46.922468 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.922265    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f8c82b7-be76-45ff-8440-eddb547acf3b-node-exporter-tls\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:46.922468 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.922303    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f8c82b7-be76-45ff-8440-eddb547acf3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:46.922468 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.922343    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsvp5\" (UniqueName: \"kubernetes.io/projected/2f8c82b7-be76-45ff-8440-eddb547acf3b-kube-api-access-jsvp5\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:46.922468 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.922373    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f8c82b7-be76-45ff-8440-eddb547acf3b-node-exporter-wtmp\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:46.922468 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:46.922398    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87a6d039-68e6-4655-bf1f-9552c7d5b789-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:47.023107 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023071    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvp5\" (UniqueName: \"kubernetes.io/projected/2f8c82b7-be76-45ff-8440-eddb547acf3b-kube-api-access-jsvp5\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.023296 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023123    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f8c82b7-be76-45ff-8440-eddb547acf3b-node-exporter-wtmp\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.023296 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023155    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87a6d039-68e6-4655-bf1f-9552c7d5b789-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:47.023296 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023201    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f8c82b7-be76-45ff-8440-eddb547acf3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.023296 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023236    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f8c82b7-be76-45ff-8440-eddb547acf3b-sys\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.023296 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023262    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87a6d039-68e6-4655-bf1f-9552c7d5b789-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:47.023296 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023293    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87a6d039-68e6-4655-bf1f-9552c7d5b789-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:47.023604 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:47.023302    2573 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 09 17:40:47.023604 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023346    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f8c82b7-be76-45ff-8440-eddb547acf3b-sys\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.023604 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:40:47.023373    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a6d039-68e6-4655-bf1f-9552c7d5b789-kube-state-metrics-tls podName:87a6d039-68e6-4655-bf1f-9552c7d5b789 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:47.523354363 +0000 UTC m=+91.536883459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/87a6d039-68e6-4655-bf1f-9552c7d5b789-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-dnq47" (UID: "87a6d039-68e6-4655-bf1f-9552c7d5b789") : secret "kube-state-metrics-tls" not found
Jul 09 17:40:47.023604 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023302    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f8c82b7-be76-45ff-8440-eddb547acf3b-node-exporter-wtmp\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.023604 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87a6d039-68e6-4655-bf1f-9552c7d5b789-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:47.023604 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f8c82b7-be76-45ff-8440-eddb547acf3b-root\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.023604 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023504    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87a6d039-68e6-4655-bf1f-9552c7d5b789-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:47.023604 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023531    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-285zb\" (UniqueName: \"kubernetes.io/projected/87a6d039-68e6-4655-bf1f-9552c7d5b789-kube-api-access-285zb\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:47.023604 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023539    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f8c82b7-be76-45ff-8440-eddb547acf3b-root\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.023604 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023580    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f8c82b7-be76-45ff-8440-eddb547acf3b-node-exporter-textfile\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.023604 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023608    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f8c82b7-be76-45ff-8440-eddb547acf3b-metrics-client-ca\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.024056 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023642    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f8c82b7-be76-45ff-8440-eddb547acf3b-node-exporter-tls\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.024056 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023673    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87a6d039-68e6-4655-bf1f-9552c7d5b789-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:47.024056 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023675    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f8c82b7-be76-45ff-8440-eddb547acf3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.024056 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.023938    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f8c82b7-be76-45ff-8440-eddb547acf3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.024306 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.024245    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f8c82b7-be76-45ff-8440-eddb547acf3b-node-exporter-textfile\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.024306 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.024279    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f8c82b7-be76-45ff-8440-eddb547acf3b-metrics-client-ca\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.024306 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.024289    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87a6d039-68e6-4655-bf1f-9552c7d5b789-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:47.024755 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.024731    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87a6d039-68e6-4655-bf1f-9552c7d5b789-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:47.026201 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.026180    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f8c82b7-be76-45ff-8440-eddb547acf3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.026385 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.026365    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f8c82b7-be76-45ff-8440-eddb547acf3b-node-exporter-tls\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.026432 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.026396    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87a6d039-68e6-4655-bf1f-9552c7d5b789-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:47.042599 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.042538    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsvp5\" (UniqueName: \"kubernetes.io/projected/2f8c82b7-be76-45ff-8440-eddb547acf3b-kube-api-access-jsvp5\") pod \"node-exporter-ms8ps\" (UID: \"2f8c82b7-be76-45ff-8440-eddb547acf3b\") " pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.046742 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.044811    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-285zb\" (UniqueName: \"kubernetes.io/projected/87a6d039-68e6-4655-bf1f-9552c7d5b789-kube-api-access-285zb\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:47.051386 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.047433    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ms8ps"
Jul 09 17:40:47.281867 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:40:47.281833    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8c82b7_be76_45ff_8440_eddb547acf3b.slice/crio-45f3754956918391dea1d1dfd066cac766b25a7a86eb6d5c231bbd12378a2577 WatchSource:0}: Error finding container 45f3754956918391dea1d1dfd066cac766b25a7a86eb6d5c231bbd12378a2577: Status 404 returned error can't find the container with id 45f3754956918391dea1d1dfd066cac766b25a7a86eb6d5c231bbd12378a2577
Jul 09 17:40:47.526329 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.526300    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87a6d039-68e6-4655-bf1f-9552c7d5b789-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:47.528613 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.528585    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87a6d039-68e6-4655-bf1f-9552c7d5b789-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-dnq47\" (UID: \"87a6d039-68e6-4655-bf1f-9552c7d5b789\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:47.653301 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.653233    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47"
Jul 09 17:40:47.831386 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:47.831354    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-dnq47"]
Jul 09 17:40:47.833935 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:40:47.833907    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a6d039_68e6_4655_bf1f_9552c7d5b789.slice/crio-8e9a3886beb4144a7611bed9872aeded30159eef53d943d8f54a57b7e13a1893 WatchSource:0}: Error finding container 8e9a3886beb4144a7611bed9872aeded30159eef53d943d8f54a57b7e13a1893: Status 404 returned error can't find the container with id 8e9a3886beb4144a7611bed9872aeded30159eef53d943d8f54a57b7e13a1893
Jul 09 17:40:48.052217 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:48.052175    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47" event={"ID":"87a6d039-68e6-4655-bf1f-9552c7d5b789","Type":"ContainerStarted","Data":"8e9a3886beb4144a7611bed9872aeded30159eef53d943d8f54a57b7e13a1893"}
Jul 09 17:40:48.055017 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:48.054771    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54d9n" event={"ID":"81dade0c-92b2-44a3-9b6d-5ba9265259a3","Type":"ContainerStarted","Data":"344697b311d10730821a3b6022f82d6708ecf504d3a8d0145146c8a5ca1f3e4f"}
Jul 09 17:40:48.056114 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:48.056090    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ms8ps" event={"ID":"2f8c82b7-be76-45ff-8440-eddb547acf3b","Type":"ContainerStarted","Data":"45f3754956918391dea1d1dfd066cac766b25a7a86eb6d5c231bbd12378a2577"}
Jul 09 17:40:49.061397 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:49.061365    2573 generic.go:356] "Generic (PLEG): container finished" podID="2f8c82b7-be76-45ff-8440-eddb547acf3b" containerID="7a932c5d5c2e5fb65150c30a34f5dcf368158719a993b3315da42be14db11c51" exitCode=0
Jul 09 17:40:49.061712 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:49.061442    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ms8ps" event={"ID":"2f8c82b7-be76-45ff-8440-eddb547acf3b","Type":"ContainerDied","Data":"7a932c5d5c2e5fb65150c30a34f5dcf368158719a993b3315da42be14db11c51"}
Jul 09 17:40:49.088185 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:49.088144    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-54d9n" podStartSLOduration=35.137993864 podStartE2EDuration="42.088129295s" totalImagesPullingTime="6.950135431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:40:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:48.088478399 +0000 UTC m=+92.102007537" watchObservedRunningTime="2026-07-09 17:40:49.088129295 +0000 UTC m=+93.101658409"
Jul 09 17:40:50.067313 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:50.067215    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ms8ps" event={"ID":"2f8c82b7-be76-45ff-8440-eddb547acf3b","Type":"ContainerStarted","Data":"0172144e556953ec258255fc6175a03014f9965977384355327c30dbfa0fcdca"}
Jul 09 17:40:50.067313 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:50.067283    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ms8ps" event={"ID":"2f8c82b7-be76-45ff-8440-eddb547acf3b","Type":"ContainerStarted","Data":"0abd86b54e0d8f87e9c41ebd65446f6d42bac6b2d84e6a02c7e65f1b4e7570a2"}
Jul 09 17:40:50.071949 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:50.071920    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47" event={"ID":"87a6d039-68e6-4655-bf1f-9552c7d5b789","Type":"ContainerStarted","Data":"8851d607eaea85cb0e7658a90635b4405ad73972864e559f84ca0a985ad51102"}
Jul 09 17:40:50.072116 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:50.071978    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47" event={"ID":"87a6d039-68e6-4655-bf1f-9552c7d5b789","Type":"ContainerStarted","Data":"8d8e002ea3c0919cf4c86b7680648806a3279b21a9d51c01e02924cbcf01d41d"}
Jul 09 17:40:50.072116 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:50.072013    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47" event={"ID":"87a6d039-68e6-4655-bf1f-9552c7d5b789","Type":"ContainerStarted","Data":"73a9e0467841d7ab406895594a6530368183c75eae226166d167f3bd87bf7bd2"}
Jul 09 17:40:50.126959 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:50.126899    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ms8ps" podStartSLOduration=3.1349811 podStartE2EDuration="4.126878028s" totalImagesPullingTime="925.350268ms" totalInitContainerRuntime="66.54666ms" isStatefulPod=true podCreationTimestamp="2026-07-09 17:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:50.093869529 +0000 UTC m=+94.107398643" watchObservedRunningTime="2026-07-09 17:40:50.126878028 +0000 UTC m=+94.140407143"
Jul 09 17:40:50.127108 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:50.127020    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-dnq47" podStartSLOduration=2.9109677940000003 podStartE2EDuration="4.127015293s" totalImagesPullingTime="1.216047499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:50.125457729 +0000 UTC m=+94.138986843" watchObservedRunningTime="2026-07-09 17:40:50.127015293 +0000 UTC m=+94.140544409"
Jul 09 17:40:56.490201 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:40:56.490166    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-558c646897-dqv7k"]
Jul 09 17:41:16.163068 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:16.163038    2573 generic.go:356] "Generic (PLEG): container finished" podID="00d27200-da1b-4d8a-9e38-65676147b344" containerID="e95b26ef568100999488881086dfe5171e9da7bbe26963e8c37275b510d1a389" exitCode=0
Jul 09 17:41:16.163415 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:16.163108    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs" event={"ID":"00d27200-da1b-4d8a-9e38-65676147b344","Type":"ContainerDied","Data":"e95b26ef568100999488881086dfe5171e9da7bbe26963e8c37275b510d1a389"}
Jul 09 17:41:16.163415 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:16.163378    2573 scope.go:117] "RemoveContainer" containerID="e95b26ef568100999488881086dfe5171e9da7bbe26963e8c37275b510d1a389"
Jul 09 17:41:17.168430 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:17.168391    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rlwgs" event={"ID":"00d27200-da1b-4d8a-9e38-65676147b344","Type":"ContainerStarted","Data":"00c25a8ebcfe53fb2db883f2021624bf0cf629140c8978e35760a175045d8ee9"}
Jul 09 17:41:21.510645 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.510575    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-558c646897-dqv7k" podUID="2e765850-1ceb-4508-a2d8-876bf7fd3a69" containerName="registry" containerID="cri-o://b7ffb2ef8180d558912f1c0da914ccbbc52974dbd9fb9534cb2b9ebfabf1468f" gracePeriod=30
Jul 09 17:41:21.745414 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.745389    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:41:21.791937 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.791854    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-bound-sa-token\") pod \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") "
Jul 09 17:41:21.791937 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.791924    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e765850-1ceb-4508-a2d8-876bf7fd3a69-ca-trust-extracted\") pod \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") "
Jul 09 17:41:21.792151 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.792093    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8brwq\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-kube-api-access-8brwq\") pod \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") "
Jul 09 17:41:21.792151 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.792132    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls\") pod \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") "
Jul 09 17:41:21.792258 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.792154    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e765850-1ceb-4508-a2d8-876bf7fd3a69-image-registry-private-configuration\") pod \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") "
Jul 09 17:41:21.792258 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.792207    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e765850-1ceb-4508-a2d8-876bf7fd3a69-installation-pull-secrets\") pod \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") "
Jul 09 17:41:21.792258 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.792226    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e765850-1ceb-4508-a2d8-876bf7fd3a69-trusted-ca\") pod \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") "
Jul 09 17:41:21.792413 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.792260    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-certificates\") pod \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\" (UID: \"2e765850-1ceb-4508-a2d8-876bf7fd3a69\") "
Jul 09 17:41:21.793683 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.792673    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e765850-1ceb-4508-a2d8-876bf7fd3a69-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2e765850-1ceb-4508-a2d8-876bf7fd3a69" (UID: "2e765850-1ceb-4508-a2d8-876bf7fd3a69"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:41:21.793683 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.792832    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2e765850-1ceb-4508-a2d8-876bf7fd3a69" (UID: "2e765850-1ceb-4508-a2d8-876bf7fd3a69"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:41:21.794579 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.794541    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-kube-api-access-8brwq" (OuterVolumeSpecName: "kube-api-access-8brwq") pod "2e765850-1ceb-4508-a2d8-876bf7fd3a69" (UID: "2e765850-1ceb-4508-a2d8-876bf7fd3a69"). InnerVolumeSpecName "kube-api-access-8brwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:41:21.794784 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.794610    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e765850-1ceb-4508-a2d8-876bf7fd3a69-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2e765850-1ceb-4508-a2d8-876bf7fd3a69" (UID: "2e765850-1ceb-4508-a2d8-876bf7fd3a69"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:41:21.794784 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.794665    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2e765850-1ceb-4508-a2d8-876bf7fd3a69" (UID: "2e765850-1ceb-4508-a2d8-876bf7fd3a69"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:41:21.794784 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.794688    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e765850-1ceb-4508-a2d8-876bf7fd3a69-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2e765850-1ceb-4508-a2d8-876bf7fd3a69" (UID: "2e765850-1ceb-4508-a2d8-876bf7fd3a69"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:41:21.794953 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.794808    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2e765850-1ceb-4508-a2d8-876bf7fd3a69" (UID: "2e765850-1ceb-4508-a2d8-876bf7fd3a69"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:41:21.800571 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.800548    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e765850-1ceb-4508-a2d8-876bf7fd3a69-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2e765850-1ceb-4508-a2d8-876bf7fd3a69" (UID: "2e765850-1ceb-4508-a2d8-876bf7fd3a69"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:41:21.893459 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.893418    2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e765850-1ceb-4508-a2d8-876bf7fd3a69-ca-trust-extracted\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:41:21.893459 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.893454    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8brwq\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-kube-api-access-8brwq\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:41:21.893459 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.893465    2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:41:21.893459 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.893475    2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e765850-1ceb-4508-a2d8-876bf7fd3a69-image-registry-private-configuration\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:41:21.893744 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.893484    2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e765850-1ceb-4508-a2d8-876bf7fd3a69-installation-pull-secrets\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:41:21.893744 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.893492    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e765850-1ceb-4508-a2d8-876bf7fd3a69-trusted-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:41:21.893744 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.893500    2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e765850-1ceb-4508-a2d8-876bf7fd3a69-registry-certificates\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:41:21.893744 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:21.893508    2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e765850-1ceb-4508-a2d8-876bf7fd3a69-bound-sa-token\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:41:22.186639 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:22.186549    2573 generic.go:356] "Generic (PLEG): container finished" podID="2e765850-1ceb-4508-a2d8-876bf7fd3a69" containerID="b7ffb2ef8180d558912f1c0da914ccbbc52974dbd9fb9534cb2b9ebfabf1468f" exitCode=0
Jul 09 17:41:22.186639 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:22.186594    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558c646897-dqv7k"
Jul 09 17:41:22.186639 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:22.186626    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558c646897-dqv7k" event={"ID":"2e765850-1ceb-4508-a2d8-876bf7fd3a69","Type":"ContainerDied","Data":"b7ffb2ef8180d558912f1c0da914ccbbc52974dbd9fb9534cb2b9ebfabf1468f"}
Jul 09 17:41:22.186937 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:22.186674    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558c646897-dqv7k" event={"ID":"2e765850-1ceb-4508-a2d8-876bf7fd3a69","Type":"ContainerDied","Data":"c7c3c98e22f4c9992d378a429bb41f2a657427a9e3898c2543ba9a5e290f4910"}
Jul 09 17:41:22.186937 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:22.186694    2573 scope.go:117] "RemoveContainer" containerID="b7ffb2ef8180d558912f1c0da914ccbbc52974dbd9fb9534cb2b9ebfabf1468f"
Jul 09 17:41:22.194966 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:22.194951    2573 scope.go:117] "RemoveContainer" containerID="b7ffb2ef8180d558912f1c0da914ccbbc52974dbd9fb9534cb2b9ebfabf1468f"
Jul 09 17:41:22.195247 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:41:22.195228    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ffb2ef8180d558912f1c0da914ccbbc52974dbd9fb9534cb2b9ebfabf1468f\": container with ID starting with b7ffb2ef8180d558912f1c0da914ccbbc52974dbd9fb9534cb2b9ebfabf1468f not found: ID does not exist" containerID="b7ffb2ef8180d558912f1c0da914ccbbc52974dbd9fb9534cb2b9ebfabf1468f"
Jul 09 17:41:22.195291 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:22.195255    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ffb2ef8180d558912f1c0da914ccbbc52974dbd9fb9534cb2b9ebfabf1468f"} err="failed to get container status \"b7ffb2ef8180d558912f1c0da914ccbbc52974dbd9fb9534cb2b9ebfabf1468f\": rpc error: code = NotFound desc = could not find container \"b7ffb2ef8180d558912f1c0da914ccbbc52974dbd9fb9534cb2b9ebfabf1468f\": container with ID starting with b7ffb2ef8180d558912f1c0da914ccbbc52974dbd9fb9534cb2b9ebfabf1468f not found: ID does not exist"
Jul 09 17:41:22.209891 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:22.209865    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-558c646897-dqv7k"]
Jul 09 17:41:22.215579 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:22.215538    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-558c646897-dqv7k"]
Jul 09 17:41:22.527694 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:22.527651    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e765850-1ceb-4508-a2d8-876bf7fd3a69" path="/var/lib/kubelet/pods/2e765850-1ceb-4508-a2d8-876bf7fd3a69/volumes"
Jul 09 17:41:25.119268 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:25.119216    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs\") pod \"network-metrics-daemon-6qvsq\" (UID: \"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc\") " pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:41:25.121633 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:25.121606    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a62897b-ea47-4b2f-8762-a4fc7b7a18dc-metrics-certs\") pod \"network-metrics-daemon-6qvsq\" (UID: \"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc\") " pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:41:25.237810 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:25.237774    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7kn9v\""
Jul 09 17:41:25.245516 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:25.245498    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qvsq"
Jul 09 17:41:25.392069 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:25.391987    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6qvsq"]
Jul 09 17:41:25.395146 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:41:25.395117    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a62897b_ea47_4b2f_8762_a4fc7b7a18dc.slice/crio-c8eb6751db33efd919a78748a4f1567b5af207ba919df9f66f68e8a875306d84 WatchSource:0}: Error finding container c8eb6751db33efd919a78748a4f1567b5af207ba919df9f66f68e8a875306d84: Status 404 returned error can't find the container with id c8eb6751db33efd919a78748a4f1567b5af207ba919df9f66f68e8a875306d84
Jul 09 17:41:26.210647 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:26.210612    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6qvsq" event={"ID":"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc","Type":"ContainerStarted","Data":"c8eb6751db33efd919a78748a4f1567b5af207ba919df9f66f68e8a875306d84"}
Jul 09 17:41:27.215678 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:27.215648    2573 generic.go:356] "Generic (PLEG): container finished" podID="5fb00f50-6d74-4c8c-9a7e-310f46e12b0f" containerID="673eef9cae209c5f632dfad034fe842759d96b0b283447616f3bb4644ed2c217" exitCode=0
Jul 09 17:41:27.216160 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:27.215742    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5" event={"ID":"5fb00f50-6d74-4c8c-9a7e-310f46e12b0f","Type":"ContainerDied","Data":"673eef9cae209c5f632dfad034fe842759d96b0b283447616f3bb4644ed2c217"}
Jul 09 17:41:27.216160 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:27.216109    2573 scope.go:117] "RemoveContainer" containerID="673eef9cae209c5f632dfad034fe842759d96b0b283447616f3bb4644ed2c217"
Jul 09 17:41:27.217553 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:27.217522    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6qvsq" event={"ID":"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc","Type":"ContainerStarted","Data":"851ee39aa76794f9bbd33061901a82d5ebc495be6cc084c370bcd234ef8f08bd"}
Jul 09 17:41:27.217641 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:27.217574    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6qvsq" event={"ID":"8a62897b-ea47-4b2f-8762-a4fc7b7a18dc","Type":"ContainerStarted","Data":"a1a601f5ef67cdbdbf4732ea898c6b8a557063ddfe7c64f799c981150d7d684f"}
Jul 09 17:41:27.253404 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:27.253341    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6qvsq" podStartSLOduration=130.282767564 podStartE2EDuration="2m11.25332077s" totalImagesPullingTime="970.553206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:27.251526056 +0000 UTC m=+131.265055191" watchObservedRunningTime="2026-07-09 17:41:27.25332077 +0000 UTC m=+131.266849885"
Jul 09 17:41:28.223208 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:28.223166    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-nmqt5" event={"ID":"5fb00f50-6d74-4c8c-9a7e-310f46e12b0f","Type":"ContainerStarted","Data":"3a930d5e7ac1c1fe883811674a06fd8e2a8de08e5d83a8eb1681758aef061286"}
Jul 09 17:41:42.276217 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:42.276186    2573 generic.go:356] "Generic (PLEG): container finished" podID="1a2bee9d-5d0f-4c62-a69b-b8e7993d858a" containerID="7e0de5ca70aa5fbc7121e86c7bef1e0518cb39f16f5fd5b2e3c6344e96b16c4e" exitCode=0
Jul 09 17:41:42.276798 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:42.276267    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j" event={"ID":"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a","Type":"ContainerDied","Data":"7e0de5ca70aa5fbc7121e86c7bef1e0518cb39f16f5fd5b2e3c6344e96b16c4e"}
Jul 09 17:41:42.276798 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:42.276611    2573 scope.go:117] "RemoveContainer" containerID="7e0de5ca70aa5fbc7121e86c7bef1e0518cb39f16f5fd5b2e3c6344e96b16c4e"
Jul 09 17:41:43.281582 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:41:43.281547    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-xkn6j" event={"ID":"1a2bee9d-5d0f-4c62-a69b-b8e7993d858a","Type":"ContainerStarted","Data":"611e4bcd517eee88ef912d117db68e3c7e66693f03063ee4667946a4c8330735"}
Jul 09 17:43:57.790144 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.790107    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"]
Jul 09 17:43:57.790700 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.790561    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e765850-1ceb-4508-a2d8-876bf7fd3a69" containerName="registry"
Jul 09 17:43:57.790700 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.790582    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e765850-1ceb-4508-a2d8-876bf7fd3a69" containerName="registry"
Jul 09 17:43:57.790700 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.790675    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e765850-1ceb-4508-a2d8-876bf7fd3a69" containerName="registry"
Jul 09 17:43:57.793860 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.793841    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.800033 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.800011    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 09 17:43:57.800301 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.800115    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 17:43:57.800421 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.800389    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 09 17:43:57.800894 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.800511    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 17:43:57.801739 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.801704    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 09 17:43:57.801823 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.801713    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 17:43:57.801823 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.801768    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 09 17:43:57.806117 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.806097    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"]
Jul 09 17:43:57.893531 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.893507    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-hub\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.893630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.893536    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxv8g\" (UniqueName: \"kubernetes.io/projected/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-kube-api-access-pxv8g\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.893630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.893555    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.893630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.893571    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-additional-service-ca\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.893773 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.893648    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.893773 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.893683    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-ca\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.994106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.994073    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-hub\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.994242 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.994115    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxv8g\" (UniqueName: \"kubernetes.io/projected/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-kube-api-access-pxv8g\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.994242 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.994143    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.994242 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.994167    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-additional-service-ca\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.994242 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.994202    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.994242 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.994236    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-ca\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.995004 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.994976    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-additional-service-ca\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.996709 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.996687    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-hub\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.997023 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.996992    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.997150 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.997134    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:57.997257 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:57.997236    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-ca\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:58.006837 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:58.006797    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxv8g\" (UniqueName: \"kubernetes.io/projected/156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce-kube-api-access-pxv8g\") pod \"cluster-proxy-proxy-agent-557bd5589b-cbt5s\" (UID: \"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:58.111491 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:58.111434    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"
Jul 09 17:43:58.235624 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:58.235594    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s"]
Jul 09 17:43:58.238585 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:43:58.238561    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod156fcab4_c3e2_432b_8bdc_bcc9c3dbb2ce.slice/crio-ac2fc3930e75b5bb6d1ccbffde75941b95afc18ed24264ea235f0087c31689ef WatchSource:0}: Error finding container ac2fc3930e75b5bb6d1ccbffde75941b95afc18ed24264ea235f0087c31689ef: Status 404 returned error can't find the container with id ac2fc3930e75b5bb6d1ccbffde75941b95afc18ed24264ea235f0087c31689ef
Jul 09 17:43:58.770787 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:43:58.770747    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s" event={"ID":"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce","Type":"ContainerStarted","Data":"ac2fc3930e75b5bb6d1ccbffde75941b95afc18ed24264ea235f0087c31689ef"}
Jul 09 17:44:04.799508 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:44:04.799412    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s" event={"ID":"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce","Type":"ContainerStarted","Data":"72f24eecfe2c9d8295fc900b8cdd3c42e57928686384afb01e08176910b328ad"}
Jul 09 17:44:04.799508 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:44:04.799463    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s" event={"ID":"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce","Type":"ContainerStarted","Data":"c08475a7560ae6cca8de90aade600b9cd234882287abaee9590c95da10f120a5"}
Jul 09 17:44:04.799508 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:44:04.799477    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s" event={"ID":"156fcab4-c3e2-432b-8bdc-bcc9c3dbb2ce","Type":"ContainerStarted","Data":"681e8fd3472fb2a59a4e2e8eff19dc70744a0c5bf302e55e7d41a0151d810450"}
Jul 09 17:44:04.821268 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:44:04.821213    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-557bd5589b-cbt5s" podStartSLOduration=1.537816149 podStartE2EDuration="7.821197204s" totalImagesPullingTime="6.283381055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:44:04.818948826 +0000 UTC m=+288.832477942" watchObservedRunningTime="2026-07-09 17:44:04.821197204 +0000 UTC m=+288.834726345"
Jul 09 17:44:16.284516 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:44:16.284488    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/1.log"
Jul 09 17:44:16.285000 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:44:16.284704    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/1.log"
Jul 09 17:44:16.292814 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:44:16.292788    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/0.log"
Jul 09 17:44:16.292973 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:44:16.292954    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/0.log"
Jul 09 17:44:16.297172 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:44:16.297155    2573 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 17:48:00.186516 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.186485    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-jxl42"]
Jul 09 17:48:00.189511 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.189495    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-jxl42"
Jul 09 17:48:00.194411 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.194385    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 09 17:48:00.195983 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.195947    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 17:48:00.195983 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.195962    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-ffzlh\""
Jul 09 17:48:00.195983 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.195954    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 17:48:00.207111 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.207083    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-jxl42"]
Jul 09 17:48:00.274486 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.274450    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkbc8\" (UniqueName: \"kubernetes.io/projected/e63e97aa-3db3-4aae-825b-926ea7955787-kube-api-access-nkbc8\") pod \"model-serving-api-fd65d7d6-jxl42\" (UID: \"e63e97aa-3db3-4aae-825b-926ea7955787\") " pod="kserve/model-serving-api-fd65d7d6-jxl42"
Jul 09 17:48:00.274660 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.274498    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e63e97aa-3db3-4aae-825b-926ea7955787-tls-certs\") pod \"model-serving-api-fd65d7d6-jxl42\" (UID: \"e63e97aa-3db3-4aae-825b-926ea7955787\") " pod="kserve/model-serving-api-fd65d7d6-jxl42"
Jul 09 17:48:00.375641 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.375600    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e63e97aa-3db3-4aae-825b-926ea7955787-tls-certs\") pod \"model-serving-api-fd65d7d6-jxl42\" (UID: \"e63e97aa-3db3-4aae-825b-926ea7955787\") " pod="kserve/model-serving-api-fd65d7d6-jxl42"
Jul 09 17:48:00.375846 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.375683    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkbc8\" (UniqueName: \"kubernetes.io/projected/e63e97aa-3db3-4aae-825b-926ea7955787-kube-api-access-nkbc8\") pod \"model-serving-api-fd65d7d6-jxl42\" (UID: \"e63e97aa-3db3-4aae-825b-926ea7955787\") " pod="kserve/model-serving-api-fd65d7d6-jxl42"
Jul 09 17:48:00.378110 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.378093    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e63e97aa-3db3-4aae-825b-926ea7955787-tls-certs\") pod \"model-serving-api-fd65d7d6-jxl42\" (UID: \"e63e97aa-3db3-4aae-825b-926ea7955787\") " pod="kserve/model-serving-api-fd65d7d6-jxl42"
Jul 09 17:48:00.388159 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.388126    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkbc8\" (UniqueName: \"kubernetes.io/projected/e63e97aa-3db3-4aae-825b-926ea7955787-kube-api-access-nkbc8\") pod \"model-serving-api-fd65d7d6-jxl42\" (UID: \"e63e97aa-3db3-4aae-825b-926ea7955787\") " pod="kserve/model-serving-api-fd65d7d6-jxl42"
Jul 09 17:48:00.499402 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.499371    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-jxl42"
Jul 09 17:48:00.642069 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.642045    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-jxl42"]
Jul 09 17:48:00.644795 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:48:00.644767    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63e97aa_3db3_4aae_825b_926ea7955787.slice/crio-1743c0371924954764ab8311b4c4aae5b5bb5e1cf0a70eb7b3757fe8a06625a9 WatchSource:0}: Error finding container 1743c0371924954764ab8311b4c4aae5b5bb5e1cf0a70eb7b3757fe8a06625a9: Status 404 returned error can't find the container with id 1743c0371924954764ab8311b4c4aae5b5bb5e1cf0a70eb7b3757fe8a06625a9
Jul 09 17:48:00.646802 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.646783    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:48:00.666242 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:00.666213    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-jxl42" event={"ID":"e63e97aa-3db3-4aae-825b-926ea7955787","Type":"ContainerStarted","Data":"1743c0371924954764ab8311b4c4aae5b5bb5e1cf0a70eb7b3757fe8a06625a9"}
Jul 09 17:48:03.681556 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:03.681519    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-jxl42" event={"ID":"e63e97aa-3db3-4aae-825b-926ea7955787","Type":"ContainerStarted","Data":"cf9da55b305449ebca4bdc13a11a8545c1ebf116fcc620d3ba2b20537034b91c"}
Jul 09 17:48:03.681959 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:03.681657    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-jxl42"
Jul 09 17:48:03.699908 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:03.699850    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-jxl42" podStartSLOduration=1.432099112 podStartE2EDuration="3.699834616s" totalImagesPullingTime="2.267735504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:48:03.699076013 +0000 UTC m=+527.712605139" watchObservedRunningTime="2026-07-09 17:48:03.699834616 +0000 UTC m=+527.713363730"
Jul 09 17:48:14.689909 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:14.689830    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-jxl42"
Jul 09 17:48:16.139775 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:16.139741    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-88p49"]
Jul 09 17:48:16.146122 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:16.146097    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-88p49"
Jul 09 17:48:16.149312 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:16.149283    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 09 17:48:16.149496 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:16.149337    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-thnpf\""
Jul 09 17:48:16.152820 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:16.152797    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-88p49"]
Jul 09 17:48:16.206059 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:16.206030    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a32118-e030-4160-ae6e-c44b733e6370-model-cache\") pod \"s3-init-88p49\" (UID: \"01a32118-e030-4160-ae6e-c44b733e6370\") " pod="kserve/s3-init-88p49"
Jul 09 17:48:16.206230 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:16.206081    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j24vj\" (UniqueName: \"kubernetes.io/projected/01a32118-e030-4160-ae6e-c44b733e6370-kube-api-access-j24vj\") pod \"s3-init-88p49\" (UID: \"01a32118-e030-4160-ae6e-c44b733e6370\") " pod="kserve/s3-init-88p49"
Jul 09 17:48:16.306478 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:16.306444    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j24vj\" (UniqueName: \"kubernetes.io/projected/01a32118-e030-4160-ae6e-c44b733e6370-kube-api-access-j24vj\") pod \"s3-init-88p49\" (UID: \"01a32118-e030-4160-ae6e-c44b733e6370\") " pod="kserve/s3-init-88p49"
Jul 09 17:48:16.306649 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:16.306589    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a32118-e030-4160-ae6e-c44b733e6370-model-cache\") pod \"s3-init-88p49\" (UID: \"01a32118-e030-4160-ae6e-c44b733e6370\") " pod="kserve/s3-init-88p49"
Jul 09 17:48:16.306934 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:16.306918    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a32118-e030-4160-ae6e-c44b733e6370-model-cache\") pod \"s3-init-88p49\" (UID: \"01a32118-e030-4160-ae6e-c44b733e6370\") " pod="kserve/s3-init-88p49"
Jul 09 17:48:16.318678 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:16.318655    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j24vj\" (UniqueName: \"kubernetes.io/projected/01a32118-e030-4160-ae6e-c44b733e6370-kube-api-access-j24vj\") pod \"s3-init-88p49\" (UID: \"01a32118-e030-4160-ae6e-c44b733e6370\") " pod="kserve/s3-init-88p49"
Jul 09 17:48:16.459634 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:16.459557    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-thnpf\""
Jul 09 17:48:16.467301 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:16.467280    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-88p49"
Jul 09 17:48:16.607130 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:16.607082    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-88p49"]
Jul 09 17:48:16.610443 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:48:16.610416    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01a32118_e030_4160_ae6e_c44b733e6370.slice/crio-9e15d75812a8fed4bdb5629ecef9f86f7b6b1016deb20b494557f8bb6eba9843 WatchSource:0}: Error finding container 9e15d75812a8fed4bdb5629ecef9f86f7b6b1016deb20b494557f8bb6eba9843: Status 404 returned error can't find the container with id 9e15d75812a8fed4bdb5629ecef9f86f7b6b1016deb20b494557f8bb6eba9843
Jul 09 17:48:16.736514 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:16.736484    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-88p49" event={"ID":"01a32118-e030-4160-ae6e-c44b733e6370","Type":"ContainerStarted","Data":"9e15d75812a8fed4bdb5629ecef9f86f7b6b1016deb20b494557f8bb6eba9843"}
Jul 09 17:48:21.759554 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:21.759509    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-88p49" event={"ID":"01a32118-e030-4160-ae6e-c44b733e6370","Type":"ContainerStarted","Data":"a337f4f956cba6b4318082fa7297969326b69811d9a8d37b21e69afb102ecd73"}
Jul 09 17:48:22.765157 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:22.765117    2573 generic.go:356] "Generic (PLEG): container finished" podID="01a32118-e030-4160-ae6e-c44b733e6370" containerID="a337f4f956cba6b4318082fa7297969326b69811d9a8d37b21e69afb102ecd73" exitCode=0
Jul 09 17:48:22.765569 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:22.765205    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-88p49" event={"ID":"01a32118-e030-4160-ae6e-c44b733e6370","Type":"ContainerDied","Data":"a337f4f956cba6b4318082fa7297969326b69811d9a8d37b21e69afb102ecd73"}
Jul 09 17:48:27.415798 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:27.415774    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 09 17:48:27.788538 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:27.788505    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-88p49" event={"ID":"01a32118-e030-4160-ae6e-c44b733e6370","Type":"ContainerStarted","Data":"0681a9a11c08b3015dd21ac714b7601278099734d7e91f2c29f63b3d21c76756"}
Jul 09 17:48:27.811256 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:48:27.811207    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-88p49" podStartSLOduration=1.564622408 podStartE2EDuration="11.811192707s" totalImagesPullingTime="8.960867027s" totalInitContainerRuntime="1.285703272s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:48:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:48:27.808513571 +0000 UTC m=+551.822042684" watchObservedRunningTime="2026-07-09 17:48:27.811192707 +0000 UTC m=+551.824721820"
Jul 09 17:49:16.308291 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:16.308259    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/1.log"
Jul 09 17:49:16.310427 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:16.310403    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/1.log"
Jul 09 17:49:16.315364 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:16.315346    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/0.log"
Jul 09 17:49:16.317662 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:16.317646    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/0.log"
Jul 09 17:49:33.029989 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:33.029957    2573 generic.go:356] "Generic (PLEG): container finished" podID="01a32118-e030-4160-ae6e-c44b733e6370" containerID="0681a9a11c08b3015dd21ac714b7601278099734d7e91f2c29f63b3d21c76756" exitCode=0
Jul 09 17:49:33.030373 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:33.030029    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-88p49" event={"ID":"01a32118-e030-4160-ae6e-c44b733e6370","Type":"ContainerDied","Data":"0681a9a11c08b3015dd21ac714b7601278099734d7e91f2c29f63b3d21c76756"}
Jul 09 17:49:34.162338 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:34.162313    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-88p49"
Jul 09 17:49:34.237141 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:34.237110    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a32118-e030-4160-ae6e-c44b733e6370-model-cache\") pod \"01a32118-e030-4160-ae6e-c44b733e6370\" (UID: \"01a32118-e030-4160-ae6e-c44b733e6370\") "
Jul 09 17:49:34.237289 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:34.237200    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j24vj\" (UniqueName: \"kubernetes.io/projected/01a32118-e030-4160-ae6e-c44b733e6370-kube-api-access-j24vj\") pod \"01a32118-e030-4160-ae6e-c44b733e6370\" (UID: \"01a32118-e030-4160-ae6e-c44b733e6370\") "
Jul 09 17:49:34.239376 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:34.239351    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a32118-e030-4160-ae6e-c44b733e6370-kube-api-access-j24vj" (OuterVolumeSpecName: "kube-api-access-j24vj") pod "01a32118-e030-4160-ae6e-c44b733e6370" (UID: "01a32118-e030-4160-ae6e-c44b733e6370"). InnerVolumeSpecName "kube-api-access-j24vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:49:34.255677 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:34.255653    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a32118-e030-4160-ae6e-c44b733e6370-model-cache" (OuterVolumeSpecName: "model-cache") pod "01a32118-e030-4160-ae6e-c44b733e6370" (UID: "01a32118-e030-4160-ae6e-c44b733e6370"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:49:34.338070 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:34.337998    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j24vj\" (UniqueName: \"kubernetes.io/projected/01a32118-e030-4160-ae6e-c44b733e6370-kube-api-access-j24vj\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:49:34.338070 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:34.338025    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a32118-e030-4160-ae6e-c44b733e6370-model-cache\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:49:35.038830 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:35.038800    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-88p49"
Jul 09 17:49:35.038991 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:35.038832    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-88p49" event={"ID":"01a32118-e030-4160-ae6e-c44b733e6370","Type":"ContainerDied","Data":"9e15d75812a8fed4bdb5629ecef9f86f7b6b1016deb20b494557f8bb6eba9843"}
Jul 09 17:49:35.038991 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:35.038870    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e15d75812a8fed4bdb5629ecef9f86f7b6b1016deb20b494557f8bb6eba9843"
Jul 09 17:49:47.182822 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.182783    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"]
Jul 09 17:49:47.183382 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.183188    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01a32118-e030-4160-ae6e-c44b733e6370" containerName="s3-init"
Jul 09 17:49:47.183382 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.183206    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="01a32118-e030-4160-ae6e-c44b733e6370" containerName="s3-init"
Jul 09 17:49:47.183382 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.183240    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01a32118-e030-4160-ae6e-c44b733e6370" containerName="download-hf-model"
Jul 09 17:49:47.183382 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.183249    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="01a32118-e030-4160-ae6e-c44b733e6370" containerName="download-hf-model"
Jul 09 17:49:47.183382 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.183335    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="01a32118-e030-4160-ae6e-c44b733e6370" containerName="s3-init"
Jul 09 17:49:47.186832 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.186812    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.190913 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.190893    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-fc881-kube-rbac-proxy-sar-config\""
Jul 09 17:49:47.191041 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.190916    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 17:49:47.191111 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.191044    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-fc881-predictor-serving-cert\""
Jul 09 17:49:47.191328 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.191312    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 17:49:47.191509 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.191495    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 17:49:47.192436 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.192395    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 17:49:47.192534 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.192473    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cfv7h\""
Jul 09 17:49:47.210327 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.210302    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"]
Jul 09 17:49:47.344393 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.344364    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.344540 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.344396    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.344540 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.344418    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.344540 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.344496    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-proxy-tls\") pod \"isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.344540 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.344516    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n2g5\" (UniqueName: \"kubernetes.io/projected/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kube-api-access-8n2g5\") pod \"isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.445837 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.445768    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-proxy-tls\") pod \"isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.445837 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.445802    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n2g5\" (UniqueName: \"kubernetes.io/projected/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kube-api-access-8n2g5\") pod \"isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.445837 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.445832    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.446084 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.445855    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.446084 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.445885    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.446353 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.446334    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.446588 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.446568    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.446630 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.446568    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.448258 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.448238    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-proxy-tls\") pod \"isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.455365 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.455341    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n2g5\" (UniqueName: \"kubernetes.io/projected/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kube-api-access-8n2g5\") pod \"isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.496451 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.496428    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:49:47.629948 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:47.629915    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"]
Jul 09 17:49:47.632876 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:49:47.632850    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19d7cea_61c3_4e82_a6f7_10b7b8fd864f.slice/crio-9d86dff015176caa027287160ee0d454dc6048ed0d2231c65993b04f60b408f6 WatchSource:0}: Error finding container 9d86dff015176caa027287160ee0d454dc6048ed0d2231c65993b04f60b408f6: Status 404 returned error can't find the container with id 9d86dff015176caa027287160ee0d454dc6048ed0d2231c65993b04f60b408f6
Jul 09 17:49:48.089168 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:48.089122    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" event={"ID":"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f","Type":"ContainerStarted","Data":"4f520db54f14cc161ae66de161fab8c9e61b676c5ec6a223c4f0fb419c9f36f6"}
Jul 09 17:49:48.089325 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:48.089251    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" event={"ID":"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f","Type":"ContainerStarted","Data":"9d86dff015176caa027287160ee0d454dc6048ed0d2231c65993b04f60b408f6"}
Jul 09 17:49:52.105851 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:52.105818    2573 generic.go:356] "Generic (PLEG): container finished" podID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerID="4f520db54f14cc161ae66de161fab8c9e61b676c5ec6a223c4f0fb419c9f36f6" exitCode=0
Jul 09 17:49:52.106351 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:49:52.105892    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" event={"ID":"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f","Type":"ContainerDied","Data":"4f520db54f14cc161ae66de161fab8c9e61b676c5ec6a223c4f0fb419c9f36f6"}
Jul 09 17:50:26.257636 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:26.257597    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" event={"ID":"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f","Type":"ContainerStarted","Data":"b825e4198043c8e4bc93ca44e5bb979e532c87b1858d09c72fef4180829bfd10"}
Jul 09 17:50:29.272232 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:29.272198    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" event={"ID":"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f","Type":"ContainerStarted","Data":"0d56c2f60cfc36113a4f4aeb031c96f2615efdd633e0b704620a2e9c7befc160"}
Jul 09 17:50:31.283596 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:31.283560    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" event={"ID":"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f","Type":"ContainerStarted","Data":"fe551820a8c9b044dd86dd73b57320619491fed49a2b4c662f95a93af7d912b1"}
Jul 09 17:50:31.284189 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:31.283672    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:50:31.313318 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:31.313262    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podStartSLOduration=2.181288277 podStartE2EDuration="44.313242279s" totalImagesPullingTime="38.402805026s" totalInitContainerRuntime="3.729148976s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:49:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:50:31.31048664 +0000 UTC m=+675.324015778" watchObservedRunningTime="2026-07-09 17:50:31.313242279 +0000 UTC m=+675.326771393"
Jul 09 17:50:32.287933 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:32.287898    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:50:32.288379 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:32.287962    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:50:32.289212 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:32.289175    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 09 17:50:32.289895 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:32.289873    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:50:33.292030 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:33.291990    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 09 17:50:33.292481 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:33.292425    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:50:33.295791 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:33.295773    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:50:34.296004 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:34.295960    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 09 17:50:34.296435 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:34.296298    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:50:44.296829 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:44.296780    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 09 17:50:44.297267 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:44.297210    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:50:54.296500 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:54.296453    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 09 17:50:54.296991 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:50:54.296970    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:51:04.296433 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:04.296388    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 09 17:51:04.296927 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:04.296867    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:51:14.296453 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:14.296356    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 09 17:51:14.296928 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:14.296840    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:51:24.296331 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:24.296277    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 09 17:51:24.296800 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:24.296768    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:51:34.296309 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:34.296264    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 09 17:51:34.296787 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:34.296764    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:51:44.296911 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:44.296880    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:51:44.297402 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:44.297160    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:51:52.240622 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.240587    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"]
Jul 09 17:51:52.241030 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.240883    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent" containerID="cri-o://fe551820a8c9b044dd86dd73b57320619491fed49a2b4c662f95a93af7d912b1" gracePeriod=30
Jul 09 17:51:52.241030 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.240929    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kube-rbac-proxy" containerID="cri-o://0d56c2f60cfc36113a4f4aeb031c96f2615efdd633e0b704620a2e9c7befc160" gracePeriod=30
Jul 09 17:51:52.241132 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.240879    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container" containerID="cri-o://b825e4198043c8e4bc93ca44e5bb979e532c87b1858d09c72fef4180829bfd10" gracePeriod=30
Jul 09 17:51:52.392593 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.392563    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"]
Jul 09 17:51:52.396073 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.396051    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.398969 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.398940    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-02e34-predictor-serving-cert\""
Jul 09 17:51:52.399074 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.399048    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-02e34-kube-rbac-proxy-sar-config\""
Jul 09 17:51:52.409081 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.409058    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"]
Jul 09 17:51:52.476418 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.476389    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"]
Jul 09 17:51:52.479830 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.479811    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.483070 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.483046    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-02e34-predictor-serving-cert\""
Jul 09 17:51:52.485552 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.485528    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-02e34-kube-rbac-proxy-sar-config\""
Jul 09 17:51:52.493499 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.493445    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"]
Jul 09 17:51:52.538043 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.538018    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b47d39ca-aae9-4e35-b703-57e80dcaead3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.538184 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.538055    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctwqn\" (UniqueName: \"kubernetes.io/projected/b47d39ca-aae9-4e35-b703-57e80dcaead3-kube-api-access-ctwqn\") pod \"isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.538184 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.538092    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b47d39ca-aae9-4e35-b703-57e80dcaead3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.538184 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.538152    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b47d39ca-aae9-4e35-b703-57e80dcaead3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.538308 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.538200    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b47d39ca-aae9-4e35-b703-57e80dcaead3-proxy-tls\") pod \"isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.596255 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.596226    2573 generic.go:356] "Generic (PLEG): container finished" podID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerID="0d56c2f60cfc36113a4f4aeb031c96f2615efdd633e0b704620a2e9c7befc160" exitCode=2
Jul 09 17:51:52.596394 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.596282    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" event={"ID":"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f","Type":"ContainerDied","Data":"0d56c2f60cfc36113a4f4aeb031c96f2615efdd633e0b704620a2e9c7befc160"}
Jul 09 17:51:52.639372 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.639337    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctwqn\" (UniqueName: \"kubernetes.io/projected/b47d39ca-aae9-4e35-b703-57e80dcaead3-kube-api-access-ctwqn\") pod \"isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.639525 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.639377    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttm5m\" (UniqueName: \"kubernetes.io/projected/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kube-api-access-ttm5m\") pod \"isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.639525 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.639401    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.639525 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.639417    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-cabundle-cert\") pod \"isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.639525 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.639502    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b47d39ca-aae9-4e35-b703-57e80dcaead3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.639708 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.639532    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-proxy-tls\") pod \"isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.639708 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.639576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b47d39ca-aae9-4e35-b703-57e80dcaead3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.639708 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.639613    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b47d39ca-aae9-4e35-b703-57e80dcaead3-proxy-tls\") pod \"isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.639708 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.639639    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b47d39ca-aae9-4e35-b703-57e80dcaead3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.639708 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.639678    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.640046 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.639940    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b47d39ca-aae9-4e35-b703-57e80dcaead3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.640316 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.640294    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b47d39ca-aae9-4e35-b703-57e80dcaead3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.640316 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.640307    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b47d39ca-aae9-4e35-b703-57e80dcaead3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.642007 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.641990    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b47d39ca-aae9-4e35-b703-57e80dcaead3-proxy-tls\") pod \"isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.649085 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.649062    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctwqn\" (UniqueName: \"kubernetes.io/projected/b47d39ca-aae9-4e35-b703-57e80dcaead3-kube-api-access-ctwqn\") pod \"isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.706191 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.706163    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:52.740898 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.740857    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttm5m\" (UniqueName: \"kubernetes.io/projected/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kube-api-access-ttm5m\") pod \"isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.741056 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.740912    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.741056 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.740939    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-cabundle-cert\") pod \"isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.741056 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.740995    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-proxy-tls\") pod \"isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.741056 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.741055    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.741401 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.741380    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.741707 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.741676    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-cabundle-cert\") pod \"isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.741707 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.741685    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.743732 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.743659    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-proxy-tls\") pod \"isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.749657 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.749633    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttm5m\" (UniqueName: \"kubernetes.io/projected/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kube-api-access-ttm5m\") pod \"isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.790349 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.790193    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:51:52.848263 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.848231    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"]
Jul 09 17:51:52.849266 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:51:52.849230    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47d39ca_aae9_4e35_b703_57e80dcaead3.slice/crio-fbc0d2c81b4f1a92f7debeaf345a0934af294aba293811957bf57ded2e934096 WatchSource:0}: Error finding container fbc0d2c81b4f1a92f7debeaf345a0934af294aba293811957bf57ded2e934096: Status 404 returned error can't find the container with id fbc0d2c81b4f1a92f7debeaf345a0934af294aba293811957bf57ded2e934096
Jul 09 17:51:52.931409 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:52.931386    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"]
Jul 09 17:51:52.933597 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:51:52.933568    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7288eb6d_3a1c_48b3_97c0_3a62a47795a9.slice/crio-23081e0979c99535d6751aa29bdc0f361b862a2b21a580c3db10b7496877f7ac WatchSource:0}: Error finding container 23081e0979c99535d6751aa29bdc0f361b862a2b21a580c3db10b7496877f7ac: Status 404 returned error can't find the container with id 23081e0979c99535d6751aa29bdc0f361b862a2b21a580c3db10b7496877f7ac
Jul 09 17:51:53.292913 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:53.292866    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 09 17:51:53.602166 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:53.602074    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" event={"ID":"7288eb6d-3a1c-48b3-97c0-3a62a47795a9","Type":"ContainerStarted","Data":"f1b54c2447ddb6a31aa35ecf600f2405e6c57d11e33b2a239cf761156f5ae8a2"}
Jul 09 17:51:53.602166 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:53.602139    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" event={"ID":"7288eb6d-3a1c-48b3-97c0-3a62a47795a9","Type":"ContainerStarted","Data":"23081e0979c99535d6751aa29bdc0f361b862a2b21a580c3db10b7496877f7ac"}
Jul 09 17:51:53.603404 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:53.603380    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" event={"ID":"b47d39ca-aae9-4e35-b703-57e80dcaead3","Type":"ContainerStarted","Data":"e1a0ab0d87937846984d54ae0076f9042f5d0832fe26dc60bf4e1be72c71f035"}
Jul 09 17:51:53.603484 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:53.603430    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" event={"ID":"b47d39ca-aae9-4e35-b703-57e80dcaead3","Type":"ContainerStarted","Data":"fbc0d2c81b4f1a92f7debeaf345a0934af294aba293811957bf57ded2e934096"}
Jul 09 17:51:54.296712 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:54.296666    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 09 17:51:54.297182 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:54.297014    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:51:56.620284 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:56.620193    2573 generic.go:356] "Generic (PLEG): container finished" podID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerID="e1a0ab0d87937846984d54ae0076f9042f5d0832fe26dc60bf4e1be72c71f035" exitCode=0
Jul 09 17:51:56.620284 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:56.620270    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" event={"ID":"b47d39ca-aae9-4e35-b703-57e80dcaead3","Type":"ContainerDied","Data":"e1a0ab0d87937846984d54ae0076f9042f5d0832fe26dc60bf4e1be72c71f035"}
Jul 09 17:51:57.625948 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:57.625917    2573 generic.go:356] "Generic (PLEG): container finished" podID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerID="b825e4198043c8e4bc93ca44e5bb979e532c87b1858d09c72fef4180829bfd10" exitCode=0
Jul 09 17:51:57.626343 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:57.625987    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" event={"ID":"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f","Type":"ContainerDied","Data":"b825e4198043c8e4bc93ca44e5bb979e532c87b1858d09c72fef4180829bfd10"}
Jul 09 17:51:57.627508 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:57.627485    2573 generic.go:356] "Generic (PLEG): container finished" podID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerID="f1b54c2447ddb6a31aa35ecf600f2405e6c57d11e33b2a239cf761156f5ae8a2" exitCode=0
Jul 09 17:51:57.627613 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:57.627552    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" event={"ID":"7288eb6d-3a1c-48b3-97c0-3a62a47795a9","Type":"ContainerDied","Data":"f1b54c2447ddb6a31aa35ecf600f2405e6c57d11e33b2a239cf761156f5ae8a2"}
Jul 09 17:51:57.629591 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:57.629573    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" event={"ID":"b47d39ca-aae9-4e35-b703-57e80dcaead3","Type":"ContainerStarted","Data":"17b66141c3ac19da0004088cb5c6c8863994f2b7589cb2148d9a7bce70f4921a"}
Jul 09 17:51:57.629638 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:57.629620    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" event={"ID":"b47d39ca-aae9-4e35-b703-57e80dcaead3","Type":"ContainerStarted","Data":"f3da434ee3a4f951e3f3aacd9dd42b26e173ed1b5525f56fb5a2e5c87636ae18"}
Jul 09 17:51:57.629900 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:57.629878    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:57.629999 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:57.629925    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:51:57.631095 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:57.631067    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 09 17:51:57.669707 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:57.669667    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" podStartSLOduration=2.228837651 podStartE2EDuration="5.669654085s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.440816434s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:51:57.669040575 +0000 UTC m=+761.682569713" watchObservedRunningTime="2026-07-09 17:51:57.669654085 +0000 UTC m=+761.683183199"
Jul 09 17:51:58.292187 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:58.292140    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 09 17:51:58.634995 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:51:58.634906    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 09 17:52:03.292578 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:03.292531    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 09 17:52:03.293048 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:03.292689    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:52:03.639756 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:03.639657    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:52:03.640184 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:03.640148    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 09 17:52:04.296332 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:04.296272    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 09 17:52:04.296798 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:04.296687    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:52:08.292078 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:08.292037    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 09 17:52:13.292515 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:13.292464    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 09 17:52:13.640887 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:13.640694    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 09 17:52:14.296937 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:14.296889    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 09 17:52:14.297421 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:14.297051    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:52:14.297421 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:14.297315    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:52:14.297534 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:14.297439    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:52:16.717285 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:16.717200    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" event={"ID":"7288eb6d-3a1c-48b3-97c0-3a62a47795a9","Type":"ContainerStarted","Data":"8554e19447427e6ec4205c87a1a81bb0629ea171bb7b7fce92c444636ea63e64"}
Jul 09 17:52:16.717285 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:16.717270    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" event={"ID":"7288eb6d-3a1c-48b3-97c0-3a62a47795a9","Type":"ContainerStarted","Data":"684abe74dad1ee6dfd3ef93018d4ac7bf0233d3acc87ffb4d2cd521af85179bd"}
Jul 09 17:52:16.717698 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:16.717613    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:52:16.752457 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:16.752410    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" podStartSLOduration=2.410688266 podStartE2EDuration="24.752396187s" totalImagesPullingTime="18.712200956s" totalInitContainerRuntime="3.629506965s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:52:16.751204482 +0000 UTC m=+780.764733595" watchObservedRunningTime="2026-07-09 17:52:16.752396187 +0000 UTC m=+780.765925302"
Jul 09 17:52:17.721147 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:17.721113    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:52:17.722396 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:17.722365    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 09 17:52:18.292231 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:18.292190    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 09 17:52:18.725079 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:18.724989    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 09 17:52:22.743122 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:22.743087    2573 generic.go:356] "Generic (PLEG): container finished" podID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerID="fe551820a8c9b044dd86dd73b57320619491fed49a2b4c662f95a93af7d912b1" exitCode=0
Jul 09 17:52:22.743466 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:22.743148    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" event={"ID":"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f","Type":"ContainerDied","Data":"fe551820a8c9b044dd86dd73b57320619491fed49a2b4c662f95a93af7d912b1"}
Jul 09 17:52:22.915754 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:22.915713    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:52:23.001535 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.001493    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kube-rbac-proxy-sar-config\") pod \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") "
Jul 09 17:52:23.001535 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.001541    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kserve-provision-location\") pod \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") "
Jul 09 17:52:23.001793 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.001599    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-cabundle-cert\") pod \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") "
Jul 09 17:52:23.001793 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.001624    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-proxy-tls\") pod \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") "
Jul 09 17:52:23.001793 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.001665    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n2g5\" (UniqueName: \"kubernetes.io/projected/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kube-api-access-8n2g5\") pod \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\" (UID: \"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f\") "
Jul 09 17:52:23.001961 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.001932    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" (UID: "e19d7cea-61c3-4e82-a6f7-10b7b8fd864f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:52:23.002000 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.001940    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" (UID: "e19d7cea-61c3-4e82-a6f7-10b7b8fd864f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:52:23.002000 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.001947    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" (UID: "e19d7cea-61c3-4e82-a6f7-10b7b8fd864f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:52:23.003950 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.003920    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kube-api-access-8n2g5" (OuterVolumeSpecName: "kube-api-access-8n2g5") pod "e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" (UID: "e19d7cea-61c3-4e82-a6f7-10b7b8fd864f"). InnerVolumeSpecName "kube-api-access-8n2g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:52:23.004053 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.003967    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" (UID: "e19d7cea-61c3-4e82-a6f7-10b7b8fd864f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:52:23.102571 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.102539    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n2g5\" (UniqueName: \"kubernetes.io/projected/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kube-api-access-8n2g5\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:52:23.102571 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.102570    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:52:23.102791 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.102580    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-kserve-provision-location\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:52:23.102791 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.102590    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-cabundle-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:52:23.102791 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.102601    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f-proxy-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:52:23.640545 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.640508    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 09 17:52:23.729772 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.729741    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:52:23.730395 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.730370    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 09 17:52:23.749599 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.749571    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"
Jul 09 17:52:23.749599 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.749584    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9" event={"ID":"e19d7cea-61c3-4e82-a6f7-10b7b8fd864f","Type":"ContainerDied","Data":"9d86dff015176caa027287160ee0d454dc6048ed0d2231c65993b04f60b408f6"}
Jul 09 17:52:23.750063 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.749653    2573 scope.go:117] "RemoveContainer" containerID="fe551820a8c9b044dd86dd73b57320619491fed49a2b4c662f95a93af7d912b1"
Jul 09 17:52:23.758857 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.758843    2573 scope.go:117] "RemoveContainer" containerID="0d56c2f60cfc36113a4f4aeb031c96f2615efdd633e0b704620a2e9c7befc160"
Jul 09 17:52:23.766061 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.766044    2573 scope.go:117] "RemoveContainer" containerID="b825e4198043c8e4bc93ca44e5bb979e532c87b1858d09c72fef4180829bfd10"
Jul 09 17:52:23.772793 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.772773    2573 scope.go:117] "RemoveContainer" containerID="4f520db54f14cc161ae66de161fab8c9e61b676c5ec6a223c4f0fb419c9f36f6"
Jul 09 17:52:23.777301 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.777281    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"]
Jul 09 17:52:23.781179 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:23.781159    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9"]
Jul 09 17:52:24.527480 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:24.527443    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" path="/var/lib/kubelet/pods/e19d7cea-61c3-4e82-a6f7-10b7b8fd864f/volumes"
Jul 09 17:52:33.641105 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:33.641068    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 09 17:52:33.730735 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:33.730686    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 09 17:52:43.641011 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:43.640930    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 09 17:52:43.730612 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:43.730573    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 09 17:52:53.640397 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:53.640353    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 09 17:52:53.731003 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:52:53.730966    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 09 17:53:03.640681 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:03.640641    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 09 17:53:03.730685 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:03.730648    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 09 17:53:13.641409 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:13.641378    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:53:13.731068 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:13.731033    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 09 17:53:23.731432 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:23.731401    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:53:42.650320 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.650286    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"]
Jul 09 17:53:42.650795 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.650625    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kserve-container" containerID="cri-o://f3da434ee3a4f951e3f3aacd9dd42b26e173ed1b5525f56fb5a2e5c87636ae18" gracePeriod=30
Jul 09 17:53:42.650795 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.650638    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kube-rbac-proxy" containerID="cri-o://17b66141c3ac19da0004088cb5c6c8863994f2b7589cb2148d9a7bce70f4921a" gracePeriod=30
Jul 09 17:53:42.720927 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.720900    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"]
Jul 09 17:53:42.721223 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.721212    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container"
Jul 09 17:53:42.721267 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.721225    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container"
Jul 09 17:53:42.721267 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.721232    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kube-rbac-proxy"
Jul 09 17:53:42.721267 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.721237    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kube-rbac-proxy"
Jul 09 17:53:42.721267 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.721249    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent"
Jul 09 17:53:42.721267 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.721254    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent"
Jul 09 17:53:42.721267 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.721268    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="storage-initializer"
Jul 09 17:53:42.721504 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.721274    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="storage-initializer"
Jul 09 17:53:42.721504 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.721319    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kube-rbac-proxy"
Jul 09 17:53:42.721504 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.721327    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="kserve-container"
Jul 09 17:53:42.721504 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.721338    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e19d7cea-61c3-4e82-a6f7-10b7b8fd864f" containerName="agent"
Jul 09 17:53:42.724558 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.724536    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:42.727403 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.727383    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-079fb-predictor-serving-cert\""
Jul 09 17:53:42.727521 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.727501    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-079fb-kube-rbac-proxy-sar-config\""
Jul 09 17:53:42.739799 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.739774    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"]
Jul 09 17:53:42.774670 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.774646    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"]
Jul 09 17:53:42.774921 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.774881    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kserve-container" containerID="cri-o://684abe74dad1ee6dfd3ef93018d4ac7bf0233d3acc87ffb4d2cd521af85179bd" gracePeriod=30
Jul 09 17:53:42.774984 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.774903    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kube-rbac-proxy" containerID="cri-o://8554e19447427e6ec4205c87a1a81bb0629ea171bb7b7fce92c444636ea63e64" gracePeriod=30
Jul 09 17:53:42.802113 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.802089    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"]
Jul 09 17:53:42.805541 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.805526    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:42.808567 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.808549    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-079fb-kube-rbac-proxy-sar-config\""
Jul 09 17:53:42.808659 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.808554    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-079fb-predictor-serving-cert\""
Jul 09 17:53:42.814683 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.814661    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"]
Jul 09 17:53:42.852657 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.852630    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/add774cf-4d7d-4e0a-a86f-422fa5c16871-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:42.852780 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.852663    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmzng\" (UniqueName: \"kubernetes.io/projected/add774cf-4d7d-4e0a-a86f-422fa5c16871-kube-api-access-nmzng\") pod \"isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:42.852780 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.852761    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/add774cf-4d7d-4e0a-a86f-422fa5c16871-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:42.852860 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.852816    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/add774cf-4d7d-4e0a-a86f-422fa5c16871-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:42.852860 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.852845    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/add774cf-4d7d-4e0a-a86f-422fa5c16871-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:42.953610 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.953554    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/add774cf-4d7d-4e0a-a86f-422fa5c16871-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:42.953746 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.953617    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:42.953746 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.953657    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/add774cf-4d7d-4e0a-a86f-422fa5c16871-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:42.953746 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.953684    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:42.953746 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.953713    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmzng\" (UniqueName: \"kubernetes.io/projected/add774cf-4d7d-4e0a-a86f-422fa5c16871-kube-api-access-nmzng\") pod \"isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:42.953977 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.953770    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e7a722b-3775-4c66-ab4c-af3000ef39ef-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:42.953977 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.953830    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e7a722b-3775-4c66-ab4c-af3000ef39ef-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:42.953977 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.953883    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlrmr\" (UniqueName: \"kubernetes.io/projected/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kube-api-access-tlrmr\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:42.953977 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.953918    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/add774cf-4d7d-4e0a-a86f-422fa5c16871-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:42.954181 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.954016    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/add774cf-4d7d-4e0a-a86f-422fa5c16871-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:42.954181 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.954124    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/add774cf-4d7d-4e0a-a86f-422fa5c16871-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:42.954590 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.954568    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/add774cf-4d7d-4e0a-a86f-422fa5c16871-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:42.954804 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.954786    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/add774cf-4d7d-4e0a-a86f-422fa5c16871-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:42.955931 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.955914    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/add774cf-4d7d-4e0a-a86f-422fa5c16871-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:42.962366 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:42.962340    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmzng\" (UniqueName: \"kubernetes.io/projected/add774cf-4d7d-4e0a-a86f-422fa5c16871-kube-api-access-nmzng\") pod \"isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:43.034341 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.034306    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:43.054611 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.054585    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:43.054748 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.054629    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e7a722b-3775-4c66-ab4c-af3000ef39ef-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:43.054748 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.054668    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e7a722b-3775-4c66-ab4c-af3000ef39ef-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:43.054748 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.054713    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlrmr\" (UniqueName: \"kubernetes.io/projected/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kube-api-access-tlrmr\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:43.054921 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.054809    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:43.054921 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:53:43.054897    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-079fb-predictor-serving-cert" not found
Jul 09 17:53:43.055021 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:53:43.054981    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7a722b-3775-4c66-ab4c-af3000ef39ef-proxy-tls podName:0e7a722b-3775-4c66-ab4c-af3000ef39ef nodeName:}" failed. No retries permitted until 2026-07-09 17:53:43.554960791 +0000 UTC m=+867.568489888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0e7a722b-3775-4c66-ab4c-af3000ef39ef-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" (UID: "0e7a722b-3775-4c66-ab4c-af3000ef39ef") : secret "isvc-xgboost-graph-raw-hpa-079fb-predictor-serving-cert" not found
Jul 09 17:53:43.055214 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.055193    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:43.055418 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.055399    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:43.055471 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.055397    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e7a722b-3775-4c66-ab4c-af3000ef39ef-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:43.066332 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.066296    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlrmr\" (UniqueName: \"kubernetes.io/projected/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kube-api-access-tlrmr\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:43.067835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.067807    2573 generic.go:356] "Generic (PLEG): container finished" podID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerID="8554e19447427e6ec4205c87a1a81bb0629ea171bb7b7fce92c444636ea63e64" exitCode=2
Jul 09 17:53:43.067979 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.067874    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" event={"ID":"7288eb6d-3a1c-48b3-97c0-3a62a47795a9","Type":"ContainerDied","Data":"8554e19447427e6ec4205c87a1a81bb0629ea171bb7b7fce92c444636ea63e64"}
Jul 09 17:53:43.070005 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.069979    2573 generic.go:356] "Generic (PLEG): container finished" podID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerID="17b66141c3ac19da0004088cb5c6c8863994f2b7589cb2148d9a7bce70f4921a" exitCode=2
Jul 09 17:53:43.070094 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.070045    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" event={"ID":"b47d39ca-aae9-4e35-b703-57e80dcaead3","Type":"ContainerDied","Data":"17b66141c3ac19da0004088cb5c6c8863994f2b7589cb2148d9a7bce70f4921a"}
Jul 09 17:53:43.174413 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.174385    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"]
Jul 09 17:53:43.176912 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:53:43.176882    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd774cf_4d7d_4e0a_a86f_422fa5c16871.slice/crio-2d63f4da0d85a62debd636b526e176ac70724ee90d29f10cb0eff4ae59d94691 WatchSource:0}: Error finding container 2d63f4da0d85a62debd636b526e176ac70724ee90d29f10cb0eff4ae59d94691: Status 404 returned error can't find the container with id 2d63f4da0d85a62debd636b526e176ac70724ee90d29f10cb0eff4ae59d94691
Jul 09 17:53:43.558760 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.558707    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e7a722b-3775-4c66-ab4c-af3000ef39ef-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:43.561324 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.561290    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e7a722b-3775-4c66-ab4c-af3000ef39ef-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:43.635157 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.635109    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 09 17:53:43.640550 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.640513    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 09 17:53:43.716092 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.716048    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:43.725493 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.725456    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 09 17:53:43.731097 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.731064    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 09 17:53:43.879180 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:43.879153    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"]
Jul 09 17:53:43.881422 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:53:43.881389    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7a722b_3775_4c66_ab4c_af3000ef39ef.slice/crio-f577960e6850c1b4d28239b26f16cf7474bfda99ec5cae940b68c201e4f41316 WatchSource:0}: Error finding container f577960e6850c1b4d28239b26f16cf7474bfda99ec5cae940b68c201e4f41316: Status 404 returned error can't find the container with id f577960e6850c1b4d28239b26f16cf7474bfda99ec5cae940b68c201e4f41316
Jul 09 17:53:44.075766 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:44.075710    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" event={"ID":"add774cf-4d7d-4e0a-a86f-422fa5c16871","Type":"ContainerStarted","Data":"0b5055067c8b951f0cce48f7ae44df8bbbb4c1038c2638fd5b4276571ace708d"}
Jul 09 17:53:44.075962 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:44.075787    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" event={"ID":"add774cf-4d7d-4e0a-a86f-422fa5c16871","Type":"ContainerStarted","Data":"2d63f4da0d85a62debd636b526e176ac70724ee90d29f10cb0eff4ae59d94691"}
Jul 09 17:53:44.077308 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:44.077283    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" event={"ID":"0e7a722b-3775-4c66-ab4c-af3000ef39ef","Type":"ContainerStarted","Data":"dc5c3376c340edceedbc68f93b5ebe2f034a8d8ef053ce9f5c603955be7512c4"}
Jul 09 17:53:44.077414 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:44.077339    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" event={"ID":"0e7a722b-3775-4c66-ab4c-af3000ef39ef","Type":"ContainerStarted","Data":"f577960e6850c1b4d28239b26f16cf7474bfda99ec5cae940b68c201e4f41316"}
Jul 09 17:53:46.518712 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.518689    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:53:46.584241 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.584162    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttm5m\" (UniqueName: \"kubernetes.io/projected/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kube-api-access-ttm5m\") pod \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") "
Jul 09 17:53:46.584241 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.584211    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kserve-provision-location\") pod \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") "
Jul 09 17:53:46.584241 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.584239    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kube-rbac-proxy-sar-config\") pod \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") "
Jul 09 17:53:46.584458 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.584352    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-cabundle-cert\") pod \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") "
Jul 09 17:53:46.584458 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.584434    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-proxy-tls\") pod \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\" (UID: \"7288eb6d-3a1c-48b3-97c0-3a62a47795a9\") "
Jul 09 17:53:46.584594 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.584569    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7288eb6d-3a1c-48b3-97c0-3a62a47795a9" (UID: "7288eb6d-3a1c-48b3-97c0-3a62a47795a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:53:46.584707 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.584631    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7288eb6d-3a1c-48b3-97c0-3a62a47795a9" (UID: "7288eb6d-3a1c-48b3-97c0-3a62a47795a9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:53:46.584707 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.584669    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7288eb6d-3a1c-48b3-97c0-3a62a47795a9" (UID: "7288eb6d-3a1c-48b3-97c0-3a62a47795a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:53:46.584826 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.584761    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kserve-provision-location\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:53:46.584826 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.584773    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:53:46.584826 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.584782    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-cabundle-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:53:46.586305 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.586283    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kube-api-access-ttm5m" (OuterVolumeSpecName: "kube-api-access-ttm5m") pod "7288eb6d-3a1c-48b3-97c0-3a62a47795a9" (UID: "7288eb6d-3a1c-48b3-97c0-3a62a47795a9"). InnerVolumeSpecName "kube-api-access-ttm5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:53:46.586418 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.586396    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7288eb6d-3a1c-48b3-97c0-3a62a47795a9" (UID: "7288eb6d-3a1c-48b3-97c0-3a62a47795a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:53:46.686061 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.686029    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-proxy-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:53:46.686061 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:46.686055    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttm5m\" (UniqueName: \"kubernetes.io/projected/7288eb6d-3a1c-48b3-97c0-3a62a47795a9-kube-api-access-ttm5m\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:53:47.090710 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.090675    2573 generic.go:356] "Generic (PLEG): container finished" podID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerID="0b5055067c8b951f0cce48f7ae44df8bbbb4c1038c2638fd5b4276571ace708d" exitCode=0
Jul 09 17:53:47.090911 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.090755    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" event={"ID":"add774cf-4d7d-4e0a-a86f-422fa5c16871","Type":"ContainerDied","Data":"0b5055067c8b951f0cce48f7ae44df8bbbb4c1038c2638fd5b4276571ace708d"}
Jul 09 17:53:47.092754 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.092712    2573 generic.go:356] "Generic (PLEG): container finished" podID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerID="684abe74dad1ee6dfd3ef93018d4ac7bf0233d3acc87ffb4d2cd521af85179bd" exitCode=0
Jul 09 17:53:47.092754 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.092742    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" event={"ID":"7288eb6d-3a1c-48b3-97c0-3a62a47795a9","Type":"ContainerDied","Data":"684abe74dad1ee6dfd3ef93018d4ac7bf0233d3acc87ffb4d2cd521af85179bd"}
Jul 09 17:53:47.092929 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.092783    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"
Jul 09 17:53:47.092929 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.092808    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m" event={"ID":"7288eb6d-3a1c-48b3-97c0-3a62a47795a9","Type":"ContainerDied","Data":"23081e0979c99535d6751aa29bdc0f361b862a2b21a580c3db10b7496877f7ac"}
Jul 09 17:53:47.092929 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.092849    2573 scope.go:117] "RemoveContainer" containerID="8554e19447427e6ec4205c87a1a81bb0629ea171bb7b7fce92c444636ea63e64"
Jul 09 17:53:47.104933 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.104917    2573 scope.go:117] "RemoveContainer" containerID="684abe74dad1ee6dfd3ef93018d4ac7bf0233d3acc87ffb4d2cd521af85179bd"
Jul 09 17:53:47.113470 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.113452    2573 scope.go:117] "RemoveContainer" containerID="f1b54c2447ddb6a31aa35ecf600f2405e6c57d11e33b2a239cf761156f5ae8a2"
Jul 09 17:53:47.134950 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.134930    2573 scope.go:117] "RemoveContainer" containerID="8554e19447427e6ec4205c87a1a81bb0629ea171bb7b7fce92c444636ea63e64"
Jul 09 17:53:47.135258 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:53:47.135241    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8554e19447427e6ec4205c87a1a81bb0629ea171bb7b7fce92c444636ea63e64\": container with ID starting with 8554e19447427e6ec4205c87a1a81bb0629ea171bb7b7fce92c444636ea63e64 not found: ID does not exist" containerID="8554e19447427e6ec4205c87a1a81bb0629ea171bb7b7fce92c444636ea63e64"
Jul 09 17:53:47.135338 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.135279    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8554e19447427e6ec4205c87a1a81bb0629ea171bb7b7fce92c444636ea63e64"} err="failed to get container status \"8554e19447427e6ec4205c87a1a81bb0629ea171bb7b7fce92c444636ea63e64\": rpc error: code = NotFound desc = could not find container \"8554e19447427e6ec4205c87a1a81bb0629ea171bb7b7fce92c444636ea63e64\": container with ID starting with 8554e19447427e6ec4205c87a1a81bb0629ea171bb7b7fce92c444636ea63e64 not found: ID does not exist"
Jul 09 17:53:47.135338 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.135305    2573 scope.go:117] "RemoveContainer" containerID="684abe74dad1ee6dfd3ef93018d4ac7bf0233d3acc87ffb4d2cd521af85179bd"
Jul 09 17:53:47.135567 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:53:47.135550    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684abe74dad1ee6dfd3ef93018d4ac7bf0233d3acc87ffb4d2cd521af85179bd\": container with ID starting with 684abe74dad1ee6dfd3ef93018d4ac7bf0233d3acc87ffb4d2cd521af85179bd not found: ID does not exist" containerID="684abe74dad1ee6dfd3ef93018d4ac7bf0233d3acc87ffb4d2cd521af85179bd"
Jul 09 17:53:47.135637 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.135574    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684abe74dad1ee6dfd3ef93018d4ac7bf0233d3acc87ffb4d2cd521af85179bd"} err="failed to get container status \"684abe74dad1ee6dfd3ef93018d4ac7bf0233d3acc87ffb4d2cd521af85179bd\": rpc error: code = NotFound desc = could not find container \"684abe74dad1ee6dfd3ef93018d4ac7bf0233d3acc87ffb4d2cd521af85179bd\": container with ID starting with 684abe74dad1ee6dfd3ef93018d4ac7bf0233d3acc87ffb4d2cd521af85179bd not found: ID does not exist"
Jul 09 17:53:47.135637 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.135594    2573 scope.go:117] "RemoveContainer" containerID="f1b54c2447ddb6a31aa35ecf600f2405e6c57d11e33b2a239cf761156f5ae8a2"
Jul 09 17:53:47.135874 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:53:47.135856    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b54c2447ddb6a31aa35ecf600f2405e6c57d11e33b2a239cf761156f5ae8a2\": container with ID starting with f1b54c2447ddb6a31aa35ecf600f2405e6c57d11e33b2a239cf761156f5ae8a2 not found: ID does not exist" containerID="f1b54c2447ddb6a31aa35ecf600f2405e6c57d11e33b2a239cf761156f5ae8a2"
Jul 09 17:53:47.135928 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.135879    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b54c2447ddb6a31aa35ecf600f2405e6c57d11e33b2a239cf761156f5ae8a2"} err="failed to get container status \"f1b54c2447ddb6a31aa35ecf600f2405e6c57d11e33b2a239cf761156f5ae8a2\": rpc error: code = NotFound desc = could not find container \"f1b54c2447ddb6a31aa35ecf600f2405e6c57d11e33b2a239cf761156f5ae8a2\": container with ID starting with f1b54c2447ddb6a31aa35ecf600f2405e6c57d11e33b2a239cf761156f5ae8a2 not found: ID does not exist"
Jul 09 17:53:47.145290 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.145269    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"]
Jul 09 17:53:47.154870 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.154849    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m"]
Jul 09 17:53:47.498412 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.498385    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:53:47.594923 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.594899    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b47d39ca-aae9-4e35-b703-57e80dcaead3-proxy-tls\") pod \"b47d39ca-aae9-4e35-b703-57e80dcaead3\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") "
Jul 09 17:53:47.595286 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.594976    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b47d39ca-aae9-4e35-b703-57e80dcaead3-cabundle-cert\") pod \"b47d39ca-aae9-4e35-b703-57e80dcaead3\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") "
Jul 09 17:53:47.595286 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.594996    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b47d39ca-aae9-4e35-b703-57e80dcaead3-kube-rbac-proxy-sar-config\") pod \"b47d39ca-aae9-4e35-b703-57e80dcaead3\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") "
Jul 09 17:53:47.595286 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.595020    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctwqn\" (UniqueName: \"kubernetes.io/projected/b47d39ca-aae9-4e35-b703-57e80dcaead3-kube-api-access-ctwqn\") pod \"b47d39ca-aae9-4e35-b703-57e80dcaead3\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") "
Jul 09 17:53:47.595286 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.595060    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b47d39ca-aae9-4e35-b703-57e80dcaead3-kserve-provision-location\") pod \"b47d39ca-aae9-4e35-b703-57e80dcaead3\" (UID: \"b47d39ca-aae9-4e35-b703-57e80dcaead3\") "
Jul 09 17:53:47.595444 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.595349    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b47d39ca-aae9-4e35-b703-57e80dcaead3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b47d39ca-aae9-4e35-b703-57e80dcaead3" (UID: "b47d39ca-aae9-4e35-b703-57e80dcaead3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:53:47.595444 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.595388    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b47d39ca-aae9-4e35-b703-57e80dcaead3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b47d39ca-aae9-4e35-b703-57e80dcaead3" (UID: "b47d39ca-aae9-4e35-b703-57e80dcaead3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:53:47.595444 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.595416    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47d39ca-aae9-4e35-b703-57e80dcaead3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b47d39ca-aae9-4e35-b703-57e80dcaead3" (UID: "b47d39ca-aae9-4e35-b703-57e80dcaead3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:53:47.597108 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.597084    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b47d39ca-aae9-4e35-b703-57e80dcaead3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b47d39ca-aae9-4e35-b703-57e80dcaead3" (UID: "b47d39ca-aae9-4e35-b703-57e80dcaead3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:53:47.597411 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.597392    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47d39ca-aae9-4e35-b703-57e80dcaead3-kube-api-access-ctwqn" (OuterVolumeSpecName: "kube-api-access-ctwqn") pod "b47d39ca-aae9-4e35-b703-57e80dcaead3" (UID: "b47d39ca-aae9-4e35-b703-57e80dcaead3"). InnerVolumeSpecName "kube-api-access-ctwqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:53:47.695970 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.695934    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b47d39ca-aae9-4e35-b703-57e80dcaead3-kserve-provision-location\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:53:47.695970 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.695965    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b47d39ca-aae9-4e35-b703-57e80dcaead3-proxy-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:53:47.695970 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.695977    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b47d39ca-aae9-4e35-b703-57e80dcaead3-cabundle-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:53:47.696173 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.695985    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b47d39ca-aae9-4e35-b703-57e80dcaead3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:53:47.696173 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:47.695994    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctwqn\" (UniqueName: \"kubernetes.io/projected/b47d39ca-aae9-4e35-b703-57e80dcaead3-kube-api-access-ctwqn\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.099402 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.099364    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" event={"ID":"add774cf-4d7d-4e0a-a86f-422fa5c16871","Type":"ContainerStarted","Data":"a83ca66e56ea9102633d691fe06d5ce9fd15dc60932af38453edbfb8122144e7"}
Jul 09 17:53:48.099582 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.099431    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" event={"ID":"add774cf-4d7d-4e0a-a86f-422fa5c16871","Type":"ContainerStarted","Data":"bed453e02dc200d43a0b8da3b91ae07f6766bcb951790cb2bedc0c349afd5928"}
Jul 09 17:53:48.099760 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.099742    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:48.100906 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.100882    2573 generic.go:356] "Generic (PLEG): container finished" podID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerID="dc5c3376c340edceedbc68f93b5ebe2f034a8d8ef053ce9f5c603955be7512c4" exitCode=0
Jul 09 17:53:48.100984 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.100949    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" event={"ID":"0e7a722b-3775-4c66-ab4c-af3000ef39ef","Type":"ContainerDied","Data":"dc5c3376c340edceedbc68f93b5ebe2f034a8d8ef053ce9f5c603955be7512c4"}
Jul 09 17:53:48.103682 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.103663    2573 generic.go:356] "Generic (PLEG): container finished" podID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerID="f3da434ee3a4f951e3f3aacd9dd42b26e173ed1b5525f56fb5a2e5c87636ae18" exitCode=0
Jul 09 17:53:48.103846 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.103693    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" event={"ID":"b47d39ca-aae9-4e35-b703-57e80dcaead3","Type":"ContainerDied","Data":"f3da434ee3a4f951e3f3aacd9dd42b26e173ed1b5525f56fb5a2e5c87636ae18"}
Jul 09 17:53:48.103846 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.103713    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"
Jul 09 17:53:48.103846 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.103745    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd" event={"ID":"b47d39ca-aae9-4e35-b703-57e80dcaead3","Type":"ContainerDied","Data":"fbc0d2c81b4f1a92f7debeaf345a0934af294aba293811957bf57ded2e934096"}
Jul 09 17:53:48.103846 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.103771    2573 scope.go:117] "RemoveContainer" containerID="17b66141c3ac19da0004088cb5c6c8863994f2b7589cb2148d9a7bce70f4921a"
Jul 09 17:53:48.119363 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.119342    2573 scope.go:117] "RemoveContainer" containerID="f3da434ee3a4f951e3f3aacd9dd42b26e173ed1b5525f56fb5a2e5c87636ae18"
Jul 09 17:53:48.123247 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.123197    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" podStartSLOduration=2.39327583 podStartE2EDuration="6.123179514s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.729903684s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:53:48.122514985 +0000 UTC m=+872.136044125" watchObservedRunningTime="2026-07-09 17:53:48.123179514 +0000 UTC m=+872.136708628"
Jul 09 17:53:48.127529 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.127506    2573 scope.go:117] "RemoveContainer" containerID="e1a0ab0d87937846984d54ae0076f9042f5d0832fe26dc60bf4e1be72c71f035"
Jul 09 17:53:48.137823 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.137764    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"]
Jul 09 17:53:48.140176 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.140151    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd"]
Jul 09 17:53:48.141288 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.141273    2573 scope.go:117] "RemoveContainer" containerID="17b66141c3ac19da0004088cb5c6c8863994f2b7589cb2148d9a7bce70f4921a"
Jul 09 17:53:48.141586 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:53:48.141567    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17b66141c3ac19da0004088cb5c6c8863994f2b7589cb2148d9a7bce70f4921a\": container with ID starting with 17b66141c3ac19da0004088cb5c6c8863994f2b7589cb2148d9a7bce70f4921a not found: ID does not exist" containerID="17b66141c3ac19da0004088cb5c6c8863994f2b7589cb2148d9a7bce70f4921a"
Jul 09 17:53:48.141653 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.141593    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b66141c3ac19da0004088cb5c6c8863994f2b7589cb2148d9a7bce70f4921a"} err="failed to get container status \"17b66141c3ac19da0004088cb5c6c8863994f2b7589cb2148d9a7bce70f4921a\": rpc error: code = NotFound desc = could not find container \"17b66141c3ac19da0004088cb5c6c8863994f2b7589cb2148d9a7bce70f4921a\": container with ID starting with 17b66141c3ac19da0004088cb5c6c8863994f2b7589cb2148d9a7bce70f4921a not found: ID does not exist"
Jul 09 17:53:48.141653 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.141612    2573 scope.go:117] "RemoveContainer" containerID="f3da434ee3a4f951e3f3aacd9dd42b26e173ed1b5525f56fb5a2e5c87636ae18"
Jul 09 17:53:48.141893 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:53:48.141875    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3da434ee3a4f951e3f3aacd9dd42b26e173ed1b5525f56fb5a2e5c87636ae18\": container with ID starting with f3da434ee3a4f951e3f3aacd9dd42b26e173ed1b5525f56fb5a2e5c87636ae18 not found: ID does not exist" containerID="f3da434ee3a4f951e3f3aacd9dd42b26e173ed1b5525f56fb5a2e5c87636ae18"
Jul 09 17:53:48.141952 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.141902    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3da434ee3a4f951e3f3aacd9dd42b26e173ed1b5525f56fb5a2e5c87636ae18"} err="failed to get container status \"f3da434ee3a4f951e3f3aacd9dd42b26e173ed1b5525f56fb5a2e5c87636ae18\": rpc error: code = NotFound desc = could not find container \"f3da434ee3a4f951e3f3aacd9dd42b26e173ed1b5525f56fb5a2e5c87636ae18\": container with ID starting with f3da434ee3a4f951e3f3aacd9dd42b26e173ed1b5525f56fb5a2e5c87636ae18 not found: ID does not exist"
Jul 09 17:53:48.141952 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.141925    2573 scope.go:117] "RemoveContainer" containerID="e1a0ab0d87937846984d54ae0076f9042f5d0832fe26dc60bf4e1be72c71f035"
Jul 09 17:53:48.142197 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:53:48.142180    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1a0ab0d87937846984d54ae0076f9042f5d0832fe26dc60bf4e1be72c71f035\": container with ID starting with e1a0ab0d87937846984d54ae0076f9042f5d0832fe26dc60bf4e1be72c71f035 not found: ID does not exist" containerID="e1a0ab0d87937846984d54ae0076f9042f5d0832fe26dc60bf4e1be72c71f035"
Jul 09 17:53:48.142239 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.142203    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1a0ab0d87937846984d54ae0076f9042f5d0832fe26dc60bf4e1be72c71f035"} err="failed to get container status \"e1a0ab0d87937846984d54ae0076f9042f5d0832fe26dc60bf4e1be72c71f035\": rpc error: code = NotFound desc = could not find container \"e1a0ab0d87937846984d54ae0076f9042f5d0832fe26dc60bf4e1be72c71f035\": container with ID starting with e1a0ab0d87937846984d54ae0076f9042f5d0832fe26dc60bf4e1be72c71f035 not found: ID does not exist"
Jul 09 17:53:48.528229 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.528197    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" path="/var/lib/kubelet/pods/7288eb6d-3a1c-48b3-97c0-3a62a47795a9/volumes"
Jul 09 17:53:48.528693 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:48.528680    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" path="/var/lib/kubelet/pods/b47d39ca-aae9-4e35-b703-57e80dcaead3/volumes"
Jul 09 17:53:49.111393 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:49.111357    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" event={"ID":"0e7a722b-3775-4c66-ab4c-af3000ef39ef","Type":"ContainerStarted","Data":"d222e033f45802e055f9fe50e37884d5779832748641b10fd5f65ced6291aad2"}
Jul 09 17:53:49.111879 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:49.111422    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" event={"ID":"0e7a722b-3775-4c66-ab4c-af3000ef39ef","Type":"ContainerStarted","Data":"a2699dc36abe7ff976078d5a19d849455b7fc12953af54b7767353b0e8f22afb"}
Jul 09 17:53:49.112112 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:49.111882    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:49.112112 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:49.111938    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:49.112112 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:49.111972    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:49.112654 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:49.112623    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 09 17:53:49.112764 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:49.112681    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 09 17:53:49.139326 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:49.139275    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" podStartSLOduration=3.496819055 podStartE2EDuration="7.139256008s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.642436953s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:53:49.137629003 +0000 UTC m=+873.151158117" watchObservedRunningTime="2026-07-09 17:53:49.139256008 +0000 UTC m=+873.152785124"
Jul 09 17:53:50.117037 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:50.116936    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 09 17:53:50.117409 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:50.117045    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 09 17:53:55.122306 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:55.122273    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:53:55.122825 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:55.122670    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:53:55.122825 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:55.122690    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 09 17:53:55.123212 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:53:55.123187    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 09 17:54:05.123408 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:05.123369    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 09 17:54:05.123408 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:05.123386    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 09 17:54:15.123779 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:15.123683    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 09 17:54:15.123779 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:15.123685    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 09 17:54:16.332983 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:16.332953    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/1.log"
Jul 09 17:54:16.336577 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:16.336551    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/1.log"
Jul 09 17:54:16.340232 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:16.340210    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/0.log"
Jul 09 17:54:16.343572 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:16.343554    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/0.log"
Jul 09 17:54:25.123254 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:25.123219    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 09 17:54:25.123629 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:25.123219    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 09 17:54:35.123050 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:35.123002    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 09 17:54:35.123514 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:35.123129    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 09 17:54:45.123514 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:45.123472    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 09 17:54:45.123924 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:45.123482    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 09 17:54:55.123474 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:55.123444    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:54:55.123906 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:55.123652    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 09 17:54:58.527545 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:54:58.527517    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:55:23.027666 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.027635    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"]
Jul 09 17:55:23.028152 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.027908    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kserve-container" containerID="cri-o://bed453e02dc200d43a0b8da3b91ae07f6766bcb951790cb2bedc0c349afd5928" gracePeriod=30
Jul 09 17:55:23.028152 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.027956    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kube-rbac-proxy" containerID="cri-o://a83ca66e56ea9102633d691fe06d5ce9fd15dc60932af38453edbfb8122144e7" gracePeriod=30
Jul 09 17:55:23.069365 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069337    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"]
Jul 09 17:55:23.069655 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069643    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="storage-initializer"
Jul 09 17:55:23.069700 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069657    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="storage-initializer"
Jul 09 17:55:23.069700 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069665    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kube-rbac-proxy"
Jul 09 17:55:23.069700 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069670    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kube-rbac-proxy"
Jul 09 17:55:23.069700 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069678    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kserve-container"
Jul 09 17:55:23.069700 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069683    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kserve-container"
Jul 09 17:55:23.069700 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069692    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kserve-container"
Jul 09 17:55:23.069700 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069697    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kserve-container"
Jul 09 17:55:23.069924 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069730    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="storage-initializer"
Jul 09 17:55:23.069924 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069738    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="storage-initializer"
Jul 09 17:55:23.069924 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069747    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kube-rbac-proxy"
Jul 09 17:55:23.069924 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069752    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kube-rbac-proxy"
Jul 09 17:55:23.069924 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069798    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kserve-container"
Jul 09 17:55:23.069924 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069806    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kserve-container"
Jul 09 17:55:23.069924 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069813    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47d39ca-aae9-4e35-b703-57e80dcaead3" containerName="kube-rbac-proxy"
Jul 09 17:55:23.069924 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.069819    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7288eb6d-3a1c-48b3-97c0-3a62a47795a9" containerName="kube-rbac-proxy"
Jul 09 17:55:23.072835 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.072814    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:55:23.076256 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.076232    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-1ffb5-kube-rbac-proxy-sar-config\""
Jul 09 17:55:23.076474 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.076458    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-1ffb5-predictor-serving-cert\""
Jul 09 17:55:23.082910 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.082888    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"]
Jul 09 17:55:23.142257 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.142228    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5fh7\" (UniqueName: \"kubernetes.io/projected/34b2e932-3477-4cbe-868b-f967dd800fc9-kube-api-access-d5fh7\") pod \"message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r\" (UID: \"34b2e932-3477-4cbe-868b-f967dd800fc9\") " pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:55:23.142428 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.142282    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34b2e932-3477-4cbe-868b-f967dd800fc9-proxy-tls\") pod \"message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r\" (UID: \"34b2e932-3477-4cbe-868b-f967dd800fc9\") " pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:55:23.142428 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.142362    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34b2e932-3477-4cbe-868b-f967dd800fc9-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r\" (UID: \"34b2e932-3477-4cbe-868b-f967dd800fc9\") " pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:55:23.154109 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.154080    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"]
Jul 09 17:55:23.154346 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.154314    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kserve-container" containerID="cri-o://a2699dc36abe7ff976078d5a19d849455b7fc12953af54b7767353b0e8f22afb" gracePeriod=30
Jul 09 17:55:23.154346 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.154329    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kube-rbac-proxy" containerID="cri-o://d222e033f45802e055f9fe50e37884d5779832748641b10fd5f65ced6291aad2" gracePeriod=30
Jul 09 17:55:23.243473 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.243448    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34b2e932-3477-4cbe-868b-f967dd800fc9-proxy-tls\") pod \"message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r\" (UID: \"34b2e932-3477-4cbe-868b-f967dd800fc9\") " pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:55:23.243598 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.243492    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34b2e932-3477-4cbe-868b-f967dd800fc9-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r\" (UID: \"34b2e932-3477-4cbe-868b-f967dd800fc9\") " pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:55:23.243598 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.243543    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5fh7\" (UniqueName: \"kubernetes.io/projected/34b2e932-3477-4cbe-868b-f967dd800fc9-kube-api-access-d5fh7\") pod \"message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r\" (UID: \"34b2e932-3477-4cbe-868b-f967dd800fc9\") " pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:55:23.244237 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.244218    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34b2e932-3477-4cbe-868b-f967dd800fc9-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r\" (UID: \"34b2e932-3477-4cbe-868b-f967dd800fc9\") " pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:55:23.245740 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.245709    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34b2e932-3477-4cbe-868b-f967dd800fc9-proxy-tls\") pod \"message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r\" (UID: \"34b2e932-3477-4cbe-868b-f967dd800fc9\") " pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:55:23.252623 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.252592    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5fh7\" (UniqueName: \"kubernetes.io/projected/34b2e932-3477-4cbe-868b-f967dd800fc9-kube-api-access-d5fh7\") pod \"message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r\" (UID: \"34b2e932-3477-4cbe-868b-f967dd800fc9\") " pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:55:23.383652 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.383570    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:55:23.495695 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.495665    2573 generic.go:356] "Generic (PLEG): container finished" podID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerID="a83ca66e56ea9102633d691fe06d5ce9fd15dc60932af38453edbfb8122144e7" exitCode=2
Jul 09 17:55:23.495865 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.495712    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" event={"ID":"add774cf-4d7d-4e0a-a86f-422fa5c16871","Type":"ContainerDied","Data":"a83ca66e56ea9102633d691fe06d5ce9fd15dc60932af38453edbfb8122144e7"}
Jul 09 17:55:23.497803 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.497778    2573 generic.go:356] "Generic (PLEG): container finished" podID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerID="d222e033f45802e055f9fe50e37884d5779832748641b10fd5f65ced6291aad2" exitCode=2
Jul 09 17:55:23.497905 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.497849    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" event={"ID":"0e7a722b-3775-4c66-ab4c-af3000ef39ef","Type":"ContainerDied","Data":"d222e033f45802e055f9fe50e37884d5779832748641b10fd5f65ced6291aad2"}
Jul 09 17:55:23.509188 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.509166    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"]
Jul 09 17:55:23.511562 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:55:23.511525    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b2e932_3477_4cbe_868b_f967dd800fc9.slice/crio-8847495e7fbf4a9f49e3d0f056a20f3ba7fe46fe5a17737118b9e2cbd353e90d WatchSource:0}: Error finding container 8847495e7fbf4a9f49e3d0f056a20f3ba7fe46fe5a17737118b9e2cbd353e90d: Status 404 returned error can't find the container with id 8847495e7fbf4a9f49e3d0f056a20f3ba7fe46fe5a17737118b9e2cbd353e90d
Jul 09 17:55:23.513301 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:23.513283    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:55:24.504159 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:24.504072    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r" event={"ID":"34b2e932-3477-4cbe-868b-f967dd800fc9","Type":"ContainerStarted","Data":"8847495e7fbf4a9f49e3d0f056a20f3ba7fe46fe5a17737118b9e2cbd353e90d"}
Jul 09 17:55:25.117796 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:25.117751    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 09 17:55:25.118092 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:25.118065    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 09 17:55:25.123372 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:25.123314    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 09 17:55:25.510638 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:25.510608    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r" event={"ID":"34b2e932-3477-4cbe-868b-f967dd800fc9","Type":"ContainerStarted","Data":"fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52"}
Jul 09 17:55:25.511119 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:25.510814    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:55:25.511119 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:25.510856    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r" event={"ID":"34b2e932-3477-4cbe-868b-f967dd800fc9","Type":"ContainerStarted","Data":"41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c"}
Jul 09 17:55:25.531345 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:25.531295    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r" podStartSLOduration=1.173254796 podStartE2EDuration="2.531282157s" totalImagesPullingTime="1.358027361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:55:25.529184368 +0000 UTC m=+969.542713473" watchObservedRunningTime="2026-07-09 17:55:25.531282157 +0000 UTC m=+969.544811270"
Jul 09 17:55:26.514918 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.514885    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:55:26.516866 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.516844    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:55:26.704857 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.704831    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:55:26.771976 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.771898    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kserve-provision-location\") pod \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") "
Jul 09 17:55:26.771976 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.771941    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kube-rbac-proxy-sar-config\") pod \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") "
Jul 09 17:55:26.771976 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.771971    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e7a722b-3775-4c66-ab4c-af3000ef39ef-proxy-tls\") pod \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") "
Jul 09 17:55:26.772186 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.772021    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e7a722b-3775-4c66-ab4c-af3000ef39ef-cabundle-cert\") pod \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") "
Jul 09 17:55:26.772186 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.772069    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlrmr\" (UniqueName: \"kubernetes.io/projected/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kube-api-access-tlrmr\") pod \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\" (UID: \"0e7a722b-3775-4c66-ab4c-af3000ef39ef\") "
Jul 09 17:55:26.772294 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.772229    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e7a722b-3775-4c66-ab4c-af3000ef39ef" (UID: "0e7a722b-3775-4c66-ab4c-af3000ef39ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:55:26.772294 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.772281    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e7a722b-3775-4c66-ab4c-af3000ef39ef" (UID: "0e7a722b-3775-4c66-ab4c-af3000ef39ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:55:26.772480 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.772457    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kserve-provision-location\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:55:26.772543 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.772487    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:55:26.772543 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.772529    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7a722b-3775-4c66-ab4c-af3000ef39ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e7a722b-3775-4c66-ab4c-af3000ef39ef" (UID: "0e7a722b-3775-4c66-ab4c-af3000ef39ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:55:26.774215 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.774172    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e7a722b-3775-4c66-ab4c-af3000ef39ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e7a722b-3775-4c66-ab4c-af3000ef39ef" (UID: "0e7a722b-3775-4c66-ab4c-af3000ef39ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:55:26.774215 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.774181    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kube-api-access-tlrmr" (OuterVolumeSpecName: "kube-api-access-tlrmr") pod "0e7a722b-3775-4c66-ab4c-af3000ef39ef" (UID: "0e7a722b-3775-4c66-ab4c-af3000ef39ef"). InnerVolumeSpecName "kube-api-access-tlrmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:55:26.873160 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.873117    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e7a722b-3775-4c66-ab4c-af3000ef39ef-proxy-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:55:26.873160 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.873150    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e7a722b-3775-4c66-ab4c-af3000ef39ef-cabundle-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:55:26.873160 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:26.873160    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlrmr\" (UniqueName: \"kubernetes.io/projected/0e7a722b-3775-4c66-ab4c-af3000ef39ef-kube-api-access-tlrmr\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:55:27.521488 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.521454    2573 generic.go:356] "Generic (PLEG): container finished" podID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerID="bed453e02dc200d43a0b8da3b91ae07f6766bcb951790cb2bedc0c349afd5928" exitCode=0
Jul 09 17:55:27.521857 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.521524    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" event={"ID":"add774cf-4d7d-4e0a-a86f-422fa5c16871","Type":"ContainerDied","Data":"bed453e02dc200d43a0b8da3b91ae07f6766bcb951790cb2bedc0c349afd5928"}
Jul 09 17:55:27.523738 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.523697    2573 generic.go:356] "Generic (PLEG): container finished" podID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerID="a2699dc36abe7ff976078d5a19d849455b7fc12953af54b7767353b0e8f22afb" exitCode=0
Jul 09 17:55:27.523848 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.523751    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" event={"ID":"0e7a722b-3775-4c66-ab4c-af3000ef39ef","Type":"ContainerDied","Data":"a2699dc36abe7ff976078d5a19d849455b7fc12953af54b7767353b0e8f22afb"}
Jul 09 17:55:27.523848 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.523781    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"
Jul 09 17:55:27.523848 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.523806    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q" event={"ID":"0e7a722b-3775-4c66-ab4c-af3000ef39ef","Type":"ContainerDied","Data":"f577960e6850c1b4d28239b26f16cf7474bfda99ec5cae940b68c201e4f41316"}
Jul 09 17:55:27.523964 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.523843    2573 scope.go:117] "RemoveContainer" containerID="d222e033f45802e055f9fe50e37884d5779832748641b10fd5f65ced6291aad2"
Jul 09 17:55:27.532166 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.532147    2573 scope.go:117] "RemoveContainer" containerID="a2699dc36abe7ff976078d5a19d849455b7fc12953af54b7767353b0e8f22afb"
Jul 09 17:55:27.540028 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.540012    2573 scope.go:117] "RemoveContainer" containerID="dc5c3376c340edceedbc68f93b5ebe2f034a8d8ef053ce9f5c603955be7512c4"
Jul 09 17:55:27.548182 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.548162    2573 scope.go:117] "RemoveContainer" containerID="d222e033f45802e055f9fe50e37884d5779832748641b10fd5f65ced6291aad2"
Jul 09 17:55:27.548489 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:55:27.548464    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d222e033f45802e055f9fe50e37884d5779832748641b10fd5f65ced6291aad2\": container with ID starting with d222e033f45802e055f9fe50e37884d5779832748641b10fd5f65ced6291aad2 not found: ID does not exist" containerID="d222e033f45802e055f9fe50e37884d5779832748641b10fd5f65ced6291aad2"
Jul 09 17:55:27.548569 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.548499    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d222e033f45802e055f9fe50e37884d5779832748641b10fd5f65ced6291aad2"} err="failed to get container status \"d222e033f45802e055f9fe50e37884d5779832748641b10fd5f65ced6291aad2\": rpc error: code = NotFound desc = could not find container \"d222e033f45802e055f9fe50e37884d5779832748641b10fd5f65ced6291aad2\": container with ID starting with d222e033f45802e055f9fe50e37884d5779832748641b10fd5f65ced6291aad2 not found: ID does not exist"
Jul 09 17:55:27.548569 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.548522    2573 scope.go:117] "RemoveContainer" containerID="a2699dc36abe7ff976078d5a19d849455b7fc12953af54b7767353b0e8f22afb"
Jul 09 17:55:27.548823 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:55:27.548797    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2699dc36abe7ff976078d5a19d849455b7fc12953af54b7767353b0e8f22afb\": container with ID starting with a2699dc36abe7ff976078d5a19d849455b7fc12953af54b7767353b0e8f22afb not found: ID does not exist" containerID="a2699dc36abe7ff976078d5a19d849455b7fc12953af54b7767353b0e8f22afb"
Jul 09 17:55:27.548897 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.548834    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2699dc36abe7ff976078d5a19d849455b7fc12953af54b7767353b0e8f22afb"} err="failed to get container status \"a2699dc36abe7ff976078d5a19d849455b7fc12953af54b7767353b0e8f22afb\": rpc error: code = NotFound desc = could not find container \"a2699dc36abe7ff976078d5a19d849455b7fc12953af54b7767353b0e8f22afb\": container with ID starting with a2699dc36abe7ff976078d5a19d849455b7fc12953af54b7767353b0e8f22afb not found: ID does not exist"
Jul 09 17:55:27.548897 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.548856    2573 scope.go:117] "RemoveContainer" containerID="dc5c3376c340edceedbc68f93b5ebe2f034a8d8ef053ce9f5c603955be7512c4"
Jul 09 17:55:27.549143 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:55:27.549124    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc5c3376c340edceedbc68f93b5ebe2f034a8d8ef053ce9f5c603955be7512c4\": container with ID starting with dc5c3376c340edceedbc68f93b5ebe2f034a8d8ef053ce9f5c603955be7512c4 not found: ID does not exist" containerID="dc5c3376c340edceedbc68f93b5ebe2f034a8d8ef053ce9f5c603955be7512c4"
Jul 09 17:55:27.549192 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.549150    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5c3376c340edceedbc68f93b5ebe2f034a8d8ef053ce9f5c603955be7512c4"} err="failed to get container status \"dc5c3376c340edceedbc68f93b5ebe2f034a8d8ef053ce9f5c603955be7512c4\": rpc error: code = NotFound desc = could not find container \"dc5c3376c340edceedbc68f93b5ebe2f034a8d8ef053ce9f5c603955be7512c4\": container with ID starting with dc5c3376c340edceedbc68f93b5ebe2f034a8d8ef053ce9f5c603955be7512c4 not found: ID does not exist"
Jul 09 17:55:27.549807 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.549789    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"]
Jul 09 17:55:27.555083 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.555062    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q"]
Jul 09 17:55:27.574512 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.574491    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:55:27.681610 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.681532    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/add774cf-4d7d-4e0a-a86f-422fa5c16871-kserve-provision-location\") pod \"add774cf-4d7d-4e0a-a86f-422fa5c16871\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") "
Jul 09 17:55:27.681610 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.681581    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/add774cf-4d7d-4e0a-a86f-422fa5c16871-proxy-tls\") pod \"add774cf-4d7d-4e0a-a86f-422fa5c16871\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") "
Jul 09 17:55:27.681855 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.681621    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/add774cf-4d7d-4e0a-a86f-422fa5c16871-kube-rbac-proxy-sar-config\") pod \"add774cf-4d7d-4e0a-a86f-422fa5c16871\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") "
Jul 09 17:55:27.681855 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.681673    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmzng\" (UniqueName: \"kubernetes.io/projected/add774cf-4d7d-4e0a-a86f-422fa5c16871-kube-api-access-nmzng\") pod \"add774cf-4d7d-4e0a-a86f-422fa5c16871\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") "
Jul 09 17:55:27.681855 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.681695    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/add774cf-4d7d-4e0a-a86f-422fa5c16871-cabundle-cert\") pod \"add774cf-4d7d-4e0a-a86f-422fa5c16871\" (UID: \"add774cf-4d7d-4e0a-a86f-422fa5c16871\") "
Jul 09 17:55:27.682007 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.681946    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add774cf-4d7d-4e0a-a86f-422fa5c16871-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "add774cf-4d7d-4e0a-a86f-422fa5c16871" (UID: "add774cf-4d7d-4e0a-a86f-422fa5c16871"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:55:27.682065 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.682050    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add774cf-4d7d-4e0a-a86f-422fa5c16871-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "add774cf-4d7d-4e0a-a86f-422fa5c16871" (UID: "add774cf-4d7d-4e0a-a86f-422fa5c16871"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:55:27.682065 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.682048    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add774cf-4d7d-4e0a-a86f-422fa5c16871-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "add774cf-4d7d-4e0a-a86f-422fa5c16871" (UID: "add774cf-4d7d-4e0a-a86f-422fa5c16871"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:55:27.683703 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.683679    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add774cf-4d7d-4e0a-a86f-422fa5c16871-kube-api-access-nmzng" (OuterVolumeSpecName: "kube-api-access-nmzng") pod "add774cf-4d7d-4e0a-a86f-422fa5c16871" (UID: "add774cf-4d7d-4e0a-a86f-422fa5c16871"). InnerVolumeSpecName "kube-api-access-nmzng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:55:27.683703 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.683694    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add774cf-4d7d-4e0a-a86f-422fa5c16871-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "add774cf-4d7d-4e0a-a86f-422fa5c16871" (UID: "add774cf-4d7d-4e0a-a86f-422fa5c16871"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:55:27.782479 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.782454    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/add774cf-4d7d-4e0a-a86f-422fa5c16871-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:55:27.782479 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.782476    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmzng\" (UniqueName: \"kubernetes.io/projected/add774cf-4d7d-4e0a-a86f-422fa5c16871-kube-api-access-nmzng\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:55:27.782606 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.782485    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/add774cf-4d7d-4e0a-a86f-422fa5c16871-cabundle-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:55:27.782606 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.782494    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/add774cf-4d7d-4e0a-a86f-422fa5c16871-kserve-provision-location\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:55:27.782606 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:27.782504    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/add774cf-4d7d-4e0a-a86f-422fa5c16871-proxy-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:55:28.527843 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:28.527806    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" path="/var/lib/kubelet/pods/0e7a722b-3775-4c66-ab4c-af3000ef39ef/volumes"
Jul 09 17:55:28.530274 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:28.530246    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"
Jul 09 17:55:28.530274 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:28.530247    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf" event={"ID":"add774cf-4d7d-4e0a-a86f-422fa5c16871","Type":"ContainerDied","Data":"2d63f4da0d85a62debd636b526e176ac70724ee90d29f10cb0eff4ae59d94691"}
Jul 09 17:55:28.530470 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:28.530321    2573 scope.go:117] "RemoveContainer" containerID="a83ca66e56ea9102633d691fe06d5ce9fd15dc60932af38453edbfb8122144e7"
Jul 09 17:55:28.539039 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:28.539023    2573 scope.go:117] "RemoveContainer" containerID="bed453e02dc200d43a0b8da3b91ae07f6766bcb951790cb2bedc0c349afd5928"
Jul 09 17:55:28.546137 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:28.546114    2573 scope.go:117] "RemoveContainer" containerID="0b5055067c8b951f0cce48f7ae44df8bbbb4c1038c2638fd5b4276571ace708d"
Jul 09 17:55:28.554282 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:28.554264    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"]
Jul 09 17:55:28.557214 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:28.557195    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf"]
Jul 09 17:55:30.527666 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:30.527634    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" path="/var/lib/kubelet/pods/add774cf-4d7d-4e0a-a86f-422fa5c16871/volumes"
Jul 09 17:55:33.535793 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:33.535764    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:55:43.115587 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.115513    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"]
Jul 09 17:55:43.115951 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.115858    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kube-rbac-proxy"
Jul 09 17:55:43.115951 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.115871    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kube-rbac-proxy"
Jul 09 17:55:43.115951 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.115881    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kserve-container"
Jul 09 17:55:43.115951 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.115888    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kserve-container"
Jul 09 17:55:43.115951 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.115896    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kserve-container"
Jul 09 17:55:43.115951 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.115901    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kserve-container"
Jul 09 17:55:43.115951 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.115909    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="storage-initializer"
Jul 09 17:55:43.115951 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.115914    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="storage-initializer"
Jul 09 17:55:43.115951 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.115921    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kube-rbac-proxy"
Jul 09 17:55:43.115951 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.115931    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kube-rbac-proxy"
Jul 09 17:55:43.115951 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.115939    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="storage-initializer"
Jul 09 17:55:43.115951 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.115944    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="storage-initializer"
Jul 09 17:55:43.116338 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.116003    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kube-rbac-proxy"
Jul 09 17:55:43.116338 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.116010    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kube-rbac-proxy"
Jul 09 17:55:43.116338 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.116019    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="add774cf-4d7d-4e0a-a86f-422fa5c16871" containerName="kserve-container"
Jul 09 17:55:43.116338 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.116025    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e7a722b-3775-4c66-ab4c-af3000ef39ef" containerName="kserve-container"
Jul 09 17:55:43.119239 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.119224    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.122054 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.122033    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-1ffb5-predictor-serving-cert\""
Jul 09 17:55:43.122673 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.122438    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-1ffb5-kube-rbac-proxy-sar-config\""
Jul 09 17:55:43.122673 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.122446    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 17:55:43.122673 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.122571    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 17:55:43.131335 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.131227    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"]
Jul 09 17:55:43.204792 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.204749    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-cabundle-cert\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.204792 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.204800    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.205022 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.204881    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kserve-provision-location\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.205022 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.204930    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-proxy-tls\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.205022 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.204981    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2tcr\" (UniqueName: \"kubernetes.io/projected/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kube-api-access-d2tcr\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.305761 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.305709    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-proxy-tls\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.305901 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.305772    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2tcr\" (UniqueName: \"kubernetes.io/projected/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kube-api-access-d2tcr\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.305901 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:55:43.305842    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-serving-cert: secret "isvc-logger-raw-1ffb5-predictor-serving-cert" not found
Jul 09 17:55:43.305901 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.305886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-cabundle-cert\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.305901 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:55:43.305899    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-proxy-tls podName:49aac4d7-63f6-4a9e-b578-e1e12dd043b2 nodeName:}" failed. No retries permitted until 2026-07-09 17:55:43.805883277 +0000 UTC m=+987.819412370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-proxy-tls") pod "isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" (UID: "49aac4d7-63f6-4a9e-b578-e1e12dd043b2") : secret "isvc-logger-raw-1ffb5-predictor-serving-cert" not found
Jul 09 17:55:43.306106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.305932    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.306106 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.305970    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kserve-provision-location\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.306290 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.306274    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kserve-provision-location\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.306483 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.306467    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.306883 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.306867    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-cabundle-cert\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.314622 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.314597    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2tcr\" (UniqueName: \"kubernetes.io/projected/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kube-api-access-d2tcr\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.810792 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.810751    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-proxy-tls\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:43.813094 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:43.813062    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-proxy-tls\") pod \"isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:44.034871 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:44.034831    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:44.164081 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:44.164057    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"]
Jul 09 17:55:44.166058 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:55:44.166031    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49aac4d7_63f6_4a9e_b578_e1e12dd043b2.slice/crio-24927c3978398e0667bdece9202b5f9a43fd1c39ee7f20529b5094c4422b39e6 WatchSource:0}: Error finding container 24927c3978398e0667bdece9202b5f9a43fd1c39ee7f20529b5094c4422b39e6: Status 404 returned error can't find the container with id 24927c3978398e0667bdece9202b5f9a43fd1c39ee7f20529b5094c4422b39e6
Jul 09 17:55:44.594755 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:44.594702    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" event={"ID":"49aac4d7-63f6-4a9e-b578-e1e12dd043b2","Type":"ContainerStarted","Data":"eb6998d241eeb7a9be3d2012d911a3aa7c548b46de30b6df0bb602eaae062ee0"}
Jul 09 17:55:44.594959 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:44.594800    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" event={"ID":"49aac4d7-63f6-4a9e-b578-e1e12dd043b2","Type":"ContainerStarted","Data":"24927c3978398e0667bdece9202b5f9a43fd1c39ee7f20529b5094c4422b39e6"}
Jul 09 17:55:48.612395 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:48.612365    2573 generic.go:356] "Generic (PLEG): container finished" podID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerID="eb6998d241eeb7a9be3d2012d911a3aa7c548b46de30b6df0bb602eaae062ee0" exitCode=0
Jul 09 17:55:48.612790 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:48.612423    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" event={"ID":"49aac4d7-63f6-4a9e-b578-e1e12dd043b2","Type":"ContainerDied","Data":"eb6998d241eeb7a9be3d2012d911a3aa7c548b46de30b6df0bb602eaae062ee0"}
Jul 09 17:55:49.619635 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:49.619599    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" event={"ID":"49aac4d7-63f6-4a9e-b578-e1e12dd043b2","Type":"ContainerStarted","Data":"957d14c6a4d2c1e3401ea2eca594038ceeae2fe5c031dd61d0be723313155372"}
Jul 09 17:55:49.620084 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:49.619665    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" event={"ID":"49aac4d7-63f6-4a9e-b578-e1e12dd043b2","Type":"ContainerStarted","Data":"d102c8e0ad830d81d6fd85f228b60d27be25a9ff6f8cc35af75dd891f2e688de"}
Jul 09 17:55:49.620084 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:49.619691    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" event={"ID":"49aac4d7-63f6-4a9e-b578-e1e12dd043b2","Type":"ContainerStarted","Data":"e8ac2d475c5e87aae7e050cb9903378078b2e8d4e49845211c5573e24ee82822"}
Jul 09 17:55:49.620084 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:49.619994    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:49.620084 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:49.620043    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:49.620084 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:49.620065    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:49.621613 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:49.621583    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 09 17:55:49.622240 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:49.622215    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:55:49.644972 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:49.644924    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podStartSLOduration=2.937057729 podStartE2EDuration="6.64490983s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.707852101s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:55:49.642225838 +0000 UTC m=+993.655755026" watchObservedRunningTime="2026-07-09 17:55:49.64490983 +0000 UTC m=+993.658438943"
Jul 09 17:55:50.623848 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:50.623803    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 09 17:55:50.624347 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:50.624325    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:55:55.627897 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:55.627865    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:55:55.628579 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:55.628548    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 09 17:55:55.628780 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:55:55.628755    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:56:05.628788 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:56:05.628746    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 09 17:56:05.629263 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:56:05.629239    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:56:15.628547 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:56:15.628510    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 09 17:56:15.628950 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:56:15.628677    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:56:25.629269 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:56:25.629220    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 09 17:56:25.629811 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:56:25.629670    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:56:35.628425 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:56:35.628386    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 09 17:56:35.628979 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:56:35.628953    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:56:45.628952 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:56:45.628912    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 09 17:56:45.629354 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:56:45.629316    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:56:55.628533 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:56:55.628489    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 09 17:56:55.628975 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:56:55.628713    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:56:58.523242 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:56:58.523195    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 09 17:56:58.523767 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:56:58.523485    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:57:08.528244 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:08.528172    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:57:08.528668 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:08.528271    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:57:18.143338 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.143304    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r_34b2e932-3477-4cbe-868b-f967dd800fc9/kserve-container/0.log"
Jul 09 17:57:18.346878 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.346844    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"]
Jul 09 17:57:18.347271 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.347239    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container" containerID="cri-o://e8ac2d475c5e87aae7e050cb9903378078b2e8d4e49845211c5573e24ee82822" gracePeriod=30
Jul 09 17:57:18.347366 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.347257    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent" containerID="cri-o://957d14c6a4d2c1e3401ea2eca594038ceeae2fe5c031dd61d0be723313155372" gracePeriod=30
Jul 09 17:57:18.347498 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.347444    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kube-rbac-proxy" containerID="cri-o://d102c8e0ad830d81d6fd85f228b60d27be25a9ff6f8cc35af75dd891f2e688de" gracePeriod=30
Jul 09 17:57:18.431611 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.431543    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"]
Jul 09 17:57:18.435117 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.435101    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:18.438105 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.438080    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-3cac6-predictor-serving-cert\""
Jul 09 17:57:18.438217 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.438107    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-3cac6-kube-rbac-proxy-sar-config\""
Jul 09 17:57:18.457449 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.457407    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"]
Jul 09 17:57:18.475266 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.475242    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"]
Jul 09 17:57:18.475514 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.475489    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r" podUID="34b2e932-3477-4cbe-868b-f967dd800fc9" containerName="kube-rbac-proxy" containerID="cri-o://fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52" gracePeriod=30
Jul 09 17:57:18.475598 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.475469    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r" podUID="34b2e932-3477-4cbe-868b-f967dd800fc9" containerName="kserve-container" containerID="cri-o://41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c" gracePeriod=30
Jul 09 17:57:18.523667 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.523637    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 09 17:57:18.523918 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.523887    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:57:18.532243 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.532220    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r" podUID="34b2e932-3477-4cbe-868b-f967dd800fc9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 09 17:57:18.568395 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.568367    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0eacc765-85e4-4319-88f6-88b545d56ac2-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:18.568477 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.568406    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbpmc\" (UniqueName: \"kubernetes.io/projected/0eacc765-85e4-4319-88f6-88b545d56ac2-kube-api-access-dbpmc\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:18.568477 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.568425    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eacc765-85e4-4319-88f6-88b545d56ac2-proxy-tls\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:18.568549 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.568502    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0eacc765-85e4-4319-88f6-88b545d56ac2-cabundle-cert\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:18.568549 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.568525    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0eacc765-85e4-4319-88f6-88b545d56ac2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:18.669484 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.669458    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0eacc765-85e4-4319-88f6-88b545d56ac2-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:18.669634 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.669494    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbpmc\" (UniqueName: \"kubernetes.io/projected/0eacc765-85e4-4319-88f6-88b545d56ac2-kube-api-access-dbpmc\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:18.669634 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.669513    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eacc765-85e4-4319-88f6-88b545d56ac2-proxy-tls\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:18.669760 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:57:18.669639    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-serving-cert: secret "isvc-sklearn-scale-raw-3cac6-predictor-serving-cert" not found
Jul 09 17:57:18.669760 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.669663    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0eacc765-85e4-4319-88f6-88b545d56ac2-cabundle-cert\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:18.669760 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:57:18.669701    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eacc765-85e4-4319-88f6-88b545d56ac2-proxy-tls podName:0eacc765-85e4-4319-88f6-88b545d56ac2 nodeName:}" failed. No retries permitted until 2026-07-09 17:57:19.16967937 +0000 UTC m=+1083.183208481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0eacc765-85e4-4319-88f6-88b545d56ac2-proxy-tls") pod "isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" (UID: "0eacc765-85e4-4319-88f6-88b545d56ac2") : secret "isvc-sklearn-scale-raw-3cac6-predictor-serving-cert" not found
Jul 09 17:57:18.669929 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.669771    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0eacc765-85e4-4319-88f6-88b545d56ac2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:18.669929 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.669900    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0eacc765-85e4-4319-88f6-88b545d56ac2-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:18.670376 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.670355    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0eacc765-85e4-4319-88f6-88b545d56ac2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:18.670461 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.670359    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0eacc765-85e4-4319-88f6-88b545d56ac2-cabundle-cert\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:18.679744 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.679704    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbpmc\" (UniqueName: \"kubernetes.io/projected/0eacc765-85e4-4319-88f6-88b545d56ac2-kube-api-access-dbpmc\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:18.719731 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.719694    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:57:18.871237 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.871202    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34b2e932-3477-4cbe-868b-f967dd800fc9-kube-rbac-proxy-sar-config\") pod \"34b2e932-3477-4cbe-868b-f967dd800fc9\" (UID: \"34b2e932-3477-4cbe-868b-f967dd800fc9\") "
Jul 09 17:57:18.871397 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.871271    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5fh7\" (UniqueName: \"kubernetes.io/projected/34b2e932-3477-4cbe-868b-f967dd800fc9-kube-api-access-d5fh7\") pod \"34b2e932-3477-4cbe-868b-f967dd800fc9\" (UID: \"34b2e932-3477-4cbe-868b-f967dd800fc9\") "
Jul 09 17:57:18.871397 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.871291    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34b2e932-3477-4cbe-868b-f967dd800fc9-proxy-tls\") pod \"34b2e932-3477-4cbe-868b-f967dd800fc9\" (UID: \"34b2e932-3477-4cbe-868b-f967dd800fc9\") "
Jul 09 17:57:18.871586 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.871557    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b2e932-3477-4cbe-868b-f967dd800fc9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34b2e932-3477-4cbe-868b-f967dd800fc9" (UID: "34b2e932-3477-4cbe-868b-f967dd800fc9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:57:18.873231 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.873212    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b2e932-3477-4cbe-868b-f967dd800fc9-kube-api-access-d5fh7" (OuterVolumeSpecName: "kube-api-access-d5fh7") pod "34b2e932-3477-4cbe-868b-f967dd800fc9" (UID: "34b2e932-3477-4cbe-868b-f967dd800fc9"). InnerVolumeSpecName "kube-api-access-d5fh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:57:18.873299 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.873215    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b2e932-3477-4cbe-868b-f967dd800fc9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34b2e932-3477-4cbe-868b-f967dd800fc9" (UID: "34b2e932-3477-4cbe-868b-f967dd800fc9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:57:18.971695 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.971615    2573 generic.go:356] "Generic (PLEG): container finished" podID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerID="d102c8e0ad830d81d6fd85f228b60d27be25a9ff6f8cc35af75dd891f2e688de" exitCode=2
Jul 09 17:57:18.971695 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.971654    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" event={"ID":"49aac4d7-63f6-4a9e-b578-e1e12dd043b2","Type":"ContainerDied","Data":"d102c8e0ad830d81d6fd85f228b60d27be25a9ff6f8cc35af75dd891f2e688de"}
Jul 09 17:57:18.971875 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.971780    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5fh7\" (UniqueName: \"kubernetes.io/projected/34b2e932-3477-4cbe-868b-f967dd800fc9-kube-api-access-d5fh7\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:57:18.971875 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.971797    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34b2e932-3477-4cbe-868b-f967dd800fc9-proxy-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:57:18.971875 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.971811    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34b2e932-3477-4cbe-868b-f967dd800fc9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:57:18.973209 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.973189    2573 generic.go:356] "Generic (PLEG): container finished" podID="34b2e932-3477-4cbe-868b-f967dd800fc9" containerID="fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52" exitCode=2
Jul 09 17:57:18.973261 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.973210    2573 generic.go:356] "Generic (PLEG): container finished" podID="34b2e932-3477-4cbe-868b-f967dd800fc9" containerID="41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c" exitCode=2
Jul 09 17:57:18.973261 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.973233    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"
Jul 09 17:57:18.973322 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.973236    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r" event={"ID":"34b2e932-3477-4cbe-868b-f967dd800fc9","Type":"ContainerDied","Data":"fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52"}
Jul 09 17:57:18.973389 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.973361    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r" event={"ID":"34b2e932-3477-4cbe-868b-f967dd800fc9","Type":"ContainerDied","Data":"41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c"}
Jul 09 17:57:18.973431 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.973404    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r" event={"ID":"34b2e932-3477-4cbe-868b-f967dd800fc9","Type":"ContainerDied","Data":"8847495e7fbf4a9f49e3d0f056a20f3ba7fe46fe5a17737118b9e2cbd353e90d"}
Jul 09 17:57:18.973467 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.973432    2573 scope.go:117] "RemoveContainer" containerID="fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52"
Jul 09 17:57:18.981668 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.981651    2573 scope.go:117] "RemoveContainer" containerID="41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c"
Jul 09 17:57:18.990217 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.990187    2573 scope.go:117] "RemoveContainer" containerID="fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52"
Jul 09 17:57:18.990511 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:57:18.990494    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52\": container with ID starting with fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52 not found: ID does not exist" containerID="fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52"
Jul 09 17:57:18.990584 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.990517    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52"} err="failed to get container status \"fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52\": rpc error: code = NotFound desc = could not find container \"fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52\": container with ID starting with fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52 not found: ID does not exist"
Jul 09 17:57:18.990584 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.990532    2573 scope.go:117] "RemoveContainer" containerID="41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c"
Jul 09 17:57:18.990801 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:57:18.990781    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c\": container with ID starting with 41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c not found: ID does not exist" containerID="41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c"
Jul 09 17:57:18.990855 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.990812    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c"} err="failed to get container status \"41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c\": rpc error: code = NotFound desc = could not find container \"41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c\": container with ID starting with 41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c not found: ID does not exist"
Jul 09 17:57:18.990855 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.990825    2573 scope.go:117] "RemoveContainer" containerID="fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52"
Jul 09 17:57:18.991084 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.991064    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52"} err="failed to get container status \"fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52\": rpc error: code = NotFound desc = could not find container \"fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52\": container with ID starting with fb97d3a87cd3b4ba756e8955869ebe1ee237bc70a02147ff53cb3a9146fc5c52 not found: ID does not exist"
Jul 09 17:57:18.991166 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.991087    2573 scope.go:117] "RemoveContainer" containerID="41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c"
Jul 09 17:57:18.991345 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.991323    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c"} err="failed to get container status \"41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c\": rpc error: code = NotFound desc = could not find container \"41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c\": container with ID starting with 41508c16ee746467c625f8801295016c8d8ca67ac0a5fd16ea3236b8ec61228c not found: ID does not exist"
Jul 09 17:57:18.995830 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:18.995808    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"]
Jul 09 17:57:19.001916 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:19.001895    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r"]
Jul 09 17:57:19.174448 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:19.174411    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eacc765-85e4-4319-88f6-88b545d56ac2-proxy-tls\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:19.176776 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:19.176753    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eacc765-85e4-4319-88f6-88b545d56ac2-proxy-tls\") pod \"isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:19.345413 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:19.345385    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:19.476092 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:19.476061    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"]
Jul 09 17:57:19.479146 ip-10-0-143-142 kubenswrapper[2573]: W0709 17:57:19.479121    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eacc765_85e4_4319_88f6_88b545d56ac2.slice/crio-1d94d2dc1ec7ad8fdbe608136eea5b91cdac4f6caec3d59f9c4463347f69139b WatchSource:0}: Error finding container 1d94d2dc1ec7ad8fdbe608136eea5b91cdac4f6caec3d59f9c4463347f69139b: Status 404 returned error can't find the container with id 1d94d2dc1ec7ad8fdbe608136eea5b91cdac4f6caec3d59f9c4463347f69139b
Jul 09 17:57:19.980294 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:19.980255    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" event={"ID":"0eacc765-85e4-4319-88f6-88b545d56ac2","Type":"ContainerStarted","Data":"ef01c8c71c3b14b5f8b47e463f2c5478542c67f324136cc0f951f49e95b50d65"}
Jul 09 17:57:19.980461 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:19.980324    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" event={"ID":"0eacc765-85e4-4319-88f6-88b545d56ac2","Type":"ContainerStarted","Data":"1d94d2dc1ec7ad8fdbe608136eea5b91cdac4f6caec3d59f9c4463347f69139b"}
Jul 09 17:57:20.531740 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:20.528790    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34b2e932-3477-4cbe-868b-f967dd800fc9" path="/var/lib/kubelet/pods/34b2e932-3477-4cbe-868b-f967dd800fc9/volumes"
Jul 09 17:57:20.624101 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:20.624063    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 09 17:57:22.993693 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:22.993660    2573 generic.go:356] "Generic (PLEG): container finished" podID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerID="e8ac2d475c5e87aae7e050cb9903378078b2e8d4e49845211c5573e24ee82822" exitCode=0
Jul 09 17:57:22.994074 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:22.993737    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" event={"ID":"49aac4d7-63f6-4a9e-b578-e1e12dd043b2","Type":"ContainerDied","Data":"e8ac2d475c5e87aae7e050cb9903378078b2e8d4e49845211c5573e24ee82822"}
Jul 09 17:57:23.999671 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:23.999636    2573 generic.go:356] "Generic (PLEG): container finished" podID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerID="ef01c8c71c3b14b5f8b47e463f2c5478542c67f324136cc0f951f49e95b50d65" exitCode=0
Jul 09 17:57:24.000094 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:23.999734    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" event={"ID":"0eacc765-85e4-4319-88f6-88b545d56ac2","Type":"ContainerDied","Data":"ef01c8c71c3b14b5f8b47e463f2c5478542c67f324136cc0f951f49e95b50d65"}
Jul 09 17:57:25.006952 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:25.006920    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" event={"ID":"0eacc765-85e4-4319-88f6-88b545d56ac2","Type":"ContainerStarted","Data":"28429fbf9e6afe082f524d52cf71f54eeaa31db8f69f3ce12355ab99b614f955"}
Jul 09 17:57:25.007385 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:25.006982    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" event={"ID":"0eacc765-85e4-4319-88f6-88b545d56ac2","Type":"ContainerStarted","Data":"5c3c2a35f68eed8a6ffb93f3347140559d1cb696b46e54c58b26f7c1947cce4a"}
Jul 09 17:57:25.007385 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:25.007233    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:25.007385 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:25.007367    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:25.008674 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:25.008643    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:57:25.042152 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:25.042100    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podStartSLOduration=3.410225906 podStartE2EDuration="7.042083885s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.631857979s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:57:25.040128547 +0000 UTC m=+1089.053657677" watchObservedRunningTime="2026-07-09 17:57:25.042083885 +0000 UTC m=+1089.055612999"
Jul 09 17:57:25.624854 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:25.624810    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 09 17:57:26.011254 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:26.011213    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:57:28.523317 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:28.523265    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 09 17:57:28.523839 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:28.523619    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:57:30.624807 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:30.624772    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 09 17:57:30.625155 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:30.624894    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:57:31.016203 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:31.016175    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:57:31.016831 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:31.016801    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:57:35.624775 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:35.624709    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 09 17:57:38.523910 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:38.523864    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 09 17:57:38.524319 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:38.524240    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 17:57:38.527595 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:38.527570    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:57:38.527751 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:38.527675    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:57:40.624582 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:40.624538    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 09 17:57:41.017537 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:41.017500    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:57:45.625068 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:45.625018    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 09 17:57:48.534984 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.534959    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:57:48.619912 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.619832    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kube-rbac-proxy-sar-config\") pod \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") "
Jul 09 17:57:48.619912 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.619875    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kserve-provision-location\") pod \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") "
Jul 09 17:57:48.619912 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.619919    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-cabundle-cert\") pod \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") "
Jul 09 17:57:48.620192 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.619988    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2tcr\" (UniqueName: \"kubernetes.io/projected/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kube-api-access-d2tcr\") pod \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") "
Jul 09 17:57:48.620192 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.620167    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-proxy-tls\") pod \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\" (UID: \"49aac4d7-63f6-4a9e-b578-e1e12dd043b2\") "
Jul 09 17:57:48.620307 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.620288    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "49aac4d7-63f6-4a9e-b578-e1e12dd043b2" (UID: "49aac4d7-63f6-4a9e-b578-e1e12dd043b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:57:48.620307 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.620292    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "49aac4d7-63f6-4a9e-b578-e1e12dd043b2" (UID: "49aac4d7-63f6-4a9e-b578-e1e12dd043b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:57:48.620392 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.620359    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "49aac4d7-63f6-4a9e-b578-e1e12dd043b2" (UID: "49aac4d7-63f6-4a9e-b578-e1e12dd043b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:57:48.620466 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.620452    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-cabundle-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:57:48.620527 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.620468    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:57:48.620527 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.620478    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kserve-provision-location\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:57:48.622167 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.622144    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kube-api-access-d2tcr" (OuterVolumeSpecName: "kube-api-access-d2tcr") pod "49aac4d7-63f6-4a9e-b578-e1e12dd043b2" (UID: "49aac4d7-63f6-4a9e-b578-e1e12dd043b2"). InnerVolumeSpecName "kube-api-access-d2tcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:57:48.622234 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.622211    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "49aac4d7-63f6-4a9e-b578-e1e12dd043b2" (UID: "49aac4d7-63f6-4a9e-b578-e1e12dd043b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:57:48.721047 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.721012    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2tcr\" (UniqueName: \"kubernetes.io/projected/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-kube-api-access-d2tcr\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:57:48.721047 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:48.721042    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49aac4d7-63f6-4a9e-b578-e1e12dd043b2-proxy-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 17:57:49.108649 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.108612    2573 generic.go:356] "Generic (PLEG): container finished" podID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerID="957d14c6a4d2c1e3401ea2eca594038ceeae2fe5c031dd61d0be723313155372" exitCode=137
Jul 09 17:57:49.108861 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.108685    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"
Jul 09 17:57:49.108861 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.108684    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" event={"ID":"49aac4d7-63f6-4a9e-b578-e1e12dd043b2","Type":"ContainerDied","Data":"957d14c6a4d2c1e3401ea2eca594038ceeae2fe5c031dd61d0be723313155372"}
Jul 09 17:57:49.108861 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.108807    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" event={"ID":"49aac4d7-63f6-4a9e-b578-e1e12dd043b2","Type":"ContainerDied","Data":"24927c3978398e0667bdece9202b5f9a43fd1c39ee7f20529b5094c4422b39e6"}
Jul 09 17:57:49.108861 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.108852    2573 scope.go:117] "RemoveContainer" containerID="957d14c6a4d2c1e3401ea2eca594038ceeae2fe5c031dd61d0be723313155372"
Jul 09 17:57:49.117481 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.117464    2573 scope.go:117] "RemoveContainer" containerID="d102c8e0ad830d81d6fd85f228b60d27be25a9ff6f8cc35af75dd891f2e688de"
Jul 09 17:57:49.124465 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.124447    2573 scope.go:117] "RemoveContainer" containerID="e8ac2d475c5e87aae7e050cb9903378078b2e8d4e49845211c5573e24ee82822"
Jul 09 17:57:49.131459 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.131442    2573 scope.go:117] "RemoveContainer" containerID="eb6998d241eeb7a9be3d2012d911a3aa7c548b46de30b6df0bb602eaae062ee0"
Jul 09 17:57:49.132907 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.132883    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"]
Jul 09 17:57:49.137777 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.137754    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf"]
Jul 09 17:57:49.138696 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.138677    2573 scope.go:117] "RemoveContainer" containerID="957d14c6a4d2c1e3401ea2eca594038ceeae2fe5c031dd61d0be723313155372"
Jul 09 17:57:49.138991 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:57:49.138974    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"957d14c6a4d2c1e3401ea2eca594038ceeae2fe5c031dd61d0be723313155372\": container with ID starting with 957d14c6a4d2c1e3401ea2eca594038ceeae2fe5c031dd61d0be723313155372 not found: ID does not exist" containerID="957d14c6a4d2c1e3401ea2eca594038ceeae2fe5c031dd61d0be723313155372"
Jul 09 17:57:49.139048 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.138998    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957d14c6a4d2c1e3401ea2eca594038ceeae2fe5c031dd61d0be723313155372"} err="failed to get container status \"957d14c6a4d2c1e3401ea2eca594038ceeae2fe5c031dd61d0be723313155372\": rpc error: code = NotFound desc = could not find container \"957d14c6a4d2c1e3401ea2eca594038ceeae2fe5c031dd61d0be723313155372\": container with ID starting with 957d14c6a4d2c1e3401ea2eca594038ceeae2fe5c031dd61d0be723313155372 not found: ID does not exist"
Jul 09 17:57:49.139048 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.139014    2573 scope.go:117] "RemoveContainer" containerID="d102c8e0ad830d81d6fd85f228b60d27be25a9ff6f8cc35af75dd891f2e688de"
Jul 09 17:57:49.139222 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:57:49.139207    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d102c8e0ad830d81d6fd85f228b60d27be25a9ff6f8cc35af75dd891f2e688de\": container with ID starting with d102c8e0ad830d81d6fd85f228b60d27be25a9ff6f8cc35af75dd891f2e688de not found: ID does not exist" containerID="d102c8e0ad830d81d6fd85f228b60d27be25a9ff6f8cc35af75dd891f2e688de"
Jul 09 17:57:49.139264 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.139225    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d102c8e0ad830d81d6fd85f228b60d27be25a9ff6f8cc35af75dd891f2e688de"} err="failed to get container status \"d102c8e0ad830d81d6fd85f228b60d27be25a9ff6f8cc35af75dd891f2e688de\": rpc error: code = NotFound desc = could not find container \"d102c8e0ad830d81d6fd85f228b60d27be25a9ff6f8cc35af75dd891f2e688de\": container with ID starting with d102c8e0ad830d81d6fd85f228b60d27be25a9ff6f8cc35af75dd891f2e688de not found: ID does not exist"
Jul 09 17:57:49.139264 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.139238    2573 scope.go:117] "RemoveContainer" containerID="e8ac2d475c5e87aae7e050cb9903378078b2e8d4e49845211c5573e24ee82822"
Jul 09 17:57:49.139465 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:57:49.139443    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ac2d475c5e87aae7e050cb9903378078b2e8d4e49845211c5573e24ee82822\": container with ID starting with e8ac2d475c5e87aae7e050cb9903378078b2e8d4e49845211c5573e24ee82822 not found: ID does not exist" containerID="e8ac2d475c5e87aae7e050cb9903378078b2e8d4e49845211c5573e24ee82822"
Jul 09 17:57:49.139502 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.139474    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ac2d475c5e87aae7e050cb9903378078b2e8d4e49845211c5573e24ee82822"} err="failed to get container status \"e8ac2d475c5e87aae7e050cb9903378078b2e8d4e49845211c5573e24ee82822\": rpc error: code = NotFound desc = could not find container \"e8ac2d475c5e87aae7e050cb9903378078b2e8d4e49845211c5573e24ee82822\": container with ID starting with e8ac2d475c5e87aae7e050cb9903378078b2e8d4e49845211c5573e24ee82822 not found: ID does not exist"
Jul 09 17:57:49.139502 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.139492    2573 scope.go:117] "RemoveContainer" containerID="eb6998d241eeb7a9be3d2012d911a3aa7c548b46de30b6df0bb602eaae062ee0"
Jul 09 17:57:49.139733 ip-10-0-143-142 kubenswrapper[2573]: E0709 17:57:49.139704    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6998d241eeb7a9be3d2012d911a3aa7c548b46de30b6df0bb602eaae062ee0\": container with ID starting with eb6998d241eeb7a9be3d2012d911a3aa7c548b46de30b6df0bb602eaae062ee0 not found: ID does not exist" containerID="eb6998d241eeb7a9be3d2012d911a3aa7c548b46de30b6df0bb602eaae062ee0"
Jul 09 17:57:49.139840 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.139824    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6998d241eeb7a9be3d2012d911a3aa7c548b46de30b6df0bb602eaae062ee0"} err="failed to get container status \"eb6998d241eeb7a9be3d2012d911a3aa7c548b46de30b6df0bb602eaae062ee0\": rpc error: code = NotFound desc = could not find container \"eb6998d241eeb7a9be3d2012d911a3aa7c548b46de30b6df0bb602eaae062ee0\": container with ID starting with eb6998d241eeb7a9be3d2012d911a3aa7c548b46de30b6df0bb602eaae062ee0 not found: ID does not exist"
Jul 09 17:57:49.524588 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.524549    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: i/o timeout"
Jul 09 17:57:49.524759 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:49.524611    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent" probeResult="failure" output="Get \"http://10.132.0.29:9081/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 17:57:50.527551 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:50.527508    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" path="/var/lib/kubelet/pods/49aac4d7-63f6-4a9e-b578-e1e12dd043b2/volumes"
Jul 09 17:57:51.017299 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:57:51.017264    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:58:01.016688 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:58:01.016638    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:58:11.016945 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:58:11.016905    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:58:21.017679 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:58:21.017639    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:58:31.016885 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:58:31.016845    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:58:41.017388 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:58:41.017305    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:58:51.016939 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:58:51.016904    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:58:51.523425 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:58:51.523384    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:59:01.523430 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:59:01.523392    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:59:11.524131 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:59:11.524092    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:59:16.356122 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:59:16.356091    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/1.log"
Jul 09 17:59:16.363659 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:59:16.363636    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/1.log"
Jul 09 17:59:16.366055 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:59:16.366033    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/0.log"
Jul 09 17:59:16.370680 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:59:16.370661    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/0.log"
Jul 09 17:59:21.523692 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:59:21.523653    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:59:31.524241 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:59:31.524190    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:59:41.523980 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:59:41.523935    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 17:59:51.524645 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:59:51.524610    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 17:59:58.580413 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:59:58.580382    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"]
Jul 09 17:59:58.583055 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:59:58.580610    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" containerID="cri-o://5c3c2a35f68eed8a6ffb93f3347140559d1cb696b46e54c58b26f7c1947cce4a" gracePeriod=30
Jul 09 17:59:58.583055 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:59:58.580775    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kube-rbac-proxy" containerID="cri-o://28429fbf9e6afe082f524d52cf71f54eeaa31db8f69f3ce12355ab99b614f955" gracePeriod=30
Jul 09 17:59:59.613943 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:59:59.613910    2573 generic.go:356] "Generic (PLEG): container finished" podID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerID="28429fbf9e6afe082f524d52cf71f54eeaa31db8f69f3ce12355ab99b614f955" exitCode=2
Jul 09 17:59:59.614306 ip-10-0-143-142 kubenswrapper[2573]: I0709 17:59:59.613978    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" event={"ID":"0eacc765-85e4-4319-88f6-88b545d56ac2","Type":"ContainerDied","Data":"28429fbf9e6afe082f524d52cf71f54eeaa31db8f69f3ce12355ab99b614f955"}
Jul 09 18:00:01.012312 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:01.012270    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 09 18:00:01.524205 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:01.524166    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 09 18:00:03.764556 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.764521    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"]
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.764842    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34b2e932-3477-4cbe-868b-f967dd800fc9" containerName="kserve-container"
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.764854    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b2e932-3477-4cbe-868b-f967dd800fc9" containerName="kserve-container"
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.764865    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34b2e932-3477-4cbe-868b-f967dd800fc9" containerName="kube-rbac-proxy"
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.764871    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b2e932-3477-4cbe-868b-f967dd800fc9" containerName="kube-rbac-proxy"
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.764880    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="storage-initializer"
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.764889    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="storage-initializer"
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.764898    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent"
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.764903    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent"
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.764919    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container"
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.764924    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container"
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.764932    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kube-rbac-proxy"
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.764937    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kube-rbac-proxy"
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.764989    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="34b2e932-3477-4cbe-868b-f967dd800fc9" containerName="kserve-container"
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.764998    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kube-rbac-proxy"
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.765005    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="34b2e932-3477-4cbe-868b-f967dd800fc9" containerName="kube-rbac-proxy"
Jul 09 18:00:03.765004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.765011    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="kserve-container"
Jul 09 18:00:03.765595 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.765017    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="49aac4d7-63f6-4a9e-b578-e1e12dd043b2" containerName="agent"
Jul 09 18:00:03.768201 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.768186    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:03.771057 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.771037    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-21e027-kube-rbac-proxy-sar-config\""
Jul 09 18:00:03.771173 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.771040    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-21e027-predictor-serving-cert\""
Jul 09 18:00:03.777156 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.777133    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"]
Jul 09 18:00:03.862140 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.862105    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-proxy-tls\") pod \"isvc-primary-21e027-predictor-59449cf66d-qpdn5\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") " pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:03.862310 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.862154    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-21e027-predictor-59449cf66d-qpdn5\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") " pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:03.862310 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.862181    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-cabundle-cert\") pod \"isvc-primary-21e027-predictor-59449cf66d-qpdn5\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") " pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:03.862310 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.862201    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kserve-provision-location\") pod \"isvc-primary-21e027-predictor-59449cf66d-qpdn5\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") " pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:03.862310 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.862224    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6wlv\" (UniqueName: \"kubernetes.io/projected/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kube-api-access-l6wlv\") pod \"isvc-primary-21e027-predictor-59449cf66d-qpdn5\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") " pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:03.963125 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.963083    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kserve-provision-location\") pod \"isvc-primary-21e027-predictor-59449cf66d-qpdn5\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") " pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:03.963125 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.963133    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6wlv\" (UniqueName: \"kubernetes.io/projected/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kube-api-access-l6wlv\") pod \"isvc-primary-21e027-predictor-59449cf66d-qpdn5\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") " pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:03.963433 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.963174    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-proxy-tls\") pod \"isvc-primary-21e027-predictor-59449cf66d-qpdn5\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") " pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:03.963433 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.963209    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-21e027-predictor-59449cf66d-qpdn5\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") " pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:03.963433 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.963240    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-cabundle-cert\") pod \"isvc-primary-21e027-predictor-59449cf66d-qpdn5\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") " pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:03.963577 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.963502    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kserve-provision-location\") pod \"isvc-primary-21e027-predictor-59449cf66d-qpdn5\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") " pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:03.963940 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.963913    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-21e027-predictor-59449cf66d-qpdn5\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") " pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:03.964058 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.963948    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-cabundle-cert\") pod \"isvc-primary-21e027-predictor-59449cf66d-qpdn5\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") " pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:03.965622 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.965601    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-proxy-tls\") pod \"isvc-primary-21e027-predictor-59449cf66d-qpdn5\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") " pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:03.972564 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:03.972543    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6wlv\" (UniqueName: \"kubernetes.io/projected/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kube-api-access-l6wlv\") pod \"isvc-primary-21e027-predictor-59449cf66d-qpdn5\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") " pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:04.078915 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:04.078831    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:04.209429 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:04.209401    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"]
Jul 09 18:00:04.211021 ip-10-0-143-142 kubenswrapper[2573]: W0709 18:00:04.210993    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5eaa6d_620e_4d88_b34f_a1936c79fe1c.slice/crio-5943b8b2a6295bb310ea56ea46afc4c529dd11ff00ffa7471edaa64b3103f4dd WatchSource:0}: Error finding container 5943b8b2a6295bb310ea56ea46afc4c529dd11ff00ffa7471edaa64b3103f4dd: Status 404 returned error can't find the container with id 5943b8b2a6295bb310ea56ea46afc4c529dd11ff00ffa7471edaa64b3103f4dd
Jul 09 18:00:04.636236 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:04.636196    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" event={"ID":"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c","Type":"ContainerStarted","Data":"8317eb20cb66d469fc9369026718457d61fb3489e5af3c88f4535c990cf6dc59"}
Jul 09 18:00:04.636393 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:04.636272    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" event={"ID":"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c","Type":"ContainerStarted","Data":"5943b8b2a6295bb310ea56ea46afc4c529dd11ff00ffa7471edaa64b3103f4dd"}
Jul 09 18:00:06.011742 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:06.011678    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 09 18:00:08.228943 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.228918    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 18:00:08.301032 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.300929    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0eacc765-85e4-4319-88f6-88b545d56ac2-kube-rbac-proxy-sar-config\") pod \"0eacc765-85e4-4319-88f6-88b545d56ac2\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") "
Jul 09 18:00:08.301032 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.300980    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbpmc\" (UniqueName: \"kubernetes.io/projected/0eacc765-85e4-4319-88f6-88b545d56ac2-kube-api-access-dbpmc\") pod \"0eacc765-85e4-4319-88f6-88b545d56ac2\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") "
Jul 09 18:00:08.301032 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.300998    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0eacc765-85e4-4319-88f6-88b545d56ac2-cabundle-cert\") pod \"0eacc765-85e4-4319-88f6-88b545d56ac2\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") "
Jul 09 18:00:08.301032 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.301022    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0eacc765-85e4-4319-88f6-88b545d56ac2-kserve-provision-location\") pod \"0eacc765-85e4-4319-88f6-88b545d56ac2\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") "
Jul 09 18:00:08.301346 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.301051    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eacc765-85e4-4319-88f6-88b545d56ac2-proxy-tls\") pod \"0eacc765-85e4-4319-88f6-88b545d56ac2\" (UID: \"0eacc765-85e4-4319-88f6-88b545d56ac2\") "
Jul 09 18:00:08.301424 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.301399    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eacc765-85e4-4319-88f6-88b545d56ac2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0eacc765-85e4-4319-88f6-88b545d56ac2" (UID: "0eacc765-85e4-4319-88f6-88b545d56ac2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:00:08.301487 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.301410    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eacc765-85e4-4319-88f6-88b545d56ac2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0eacc765-85e4-4319-88f6-88b545d56ac2" (UID: "0eacc765-85e4-4319-88f6-88b545d56ac2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:00:08.301487 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.301436    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eacc765-85e4-4319-88f6-88b545d56ac2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0eacc765-85e4-4319-88f6-88b545d56ac2" (UID: "0eacc765-85e4-4319-88f6-88b545d56ac2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:00:08.303266 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.303239    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eacc765-85e4-4319-88f6-88b545d56ac2-kube-api-access-dbpmc" (OuterVolumeSpecName: "kube-api-access-dbpmc") pod "0eacc765-85e4-4319-88f6-88b545d56ac2" (UID: "0eacc765-85e4-4319-88f6-88b545d56ac2"). InnerVolumeSpecName "kube-api-access-dbpmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:00:08.303350 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.303313    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eacc765-85e4-4319-88f6-88b545d56ac2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0eacc765-85e4-4319-88f6-88b545d56ac2" (UID: "0eacc765-85e4-4319-88f6-88b545d56ac2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:00:08.402499 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.402462    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0eacc765-85e4-4319-88f6-88b545d56ac2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:00:08.402499 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.402490    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbpmc\" (UniqueName: \"kubernetes.io/projected/0eacc765-85e4-4319-88f6-88b545d56ac2-kube-api-access-dbpmc\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:00:08.402499 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.402502    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0eacc765-85e4-4319-88f6-88b545d56ac2-cabundle-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:00:08.402499 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.402511    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0eacc765-85e4-4319-88f6-88b545d56ac2-kserve-provision-location\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:00:08.402808 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.402519    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eacc765-85e4-4319-88f6-88b545d56ac2-proxy-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:00:08.654139 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.654050    2573 generic.go:356] "Generic (PLEG): container finished" podID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerID="5c3c2a35f68eed8a6ffb93f3347140559d1cb696b46e54c58b26f7c1947cce4a" exitCode=0
Jul 09 18:00:08.654139 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.654106    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"
Jul 09 18:00:08.654373 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.654133    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" event={"ID":"0eacc765-85e4-4319-88f6-88b545d56ac2","Type":"ContainerDied","Data":"5c3c2a35f68eed8a6ffb93f3347140559d1cb696b46e54c58b26f7c1947cce4a"}
Jul 09 18:00:08.654373 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.654198    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9" event={"ID":"0eacc765-85e4-4319-88f6-88b545d56ac2","Type":"ContainerDied","Data":"1d94d2dc1ec7ad8fdbe608136eea5b91cdac4f6caec3d59f9c4463347f69139b"}
Jul 09 18:00:08.654373 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.654222    2573 scope.go:117] "RemoveContainer" containerID="28429fbf9e6afe082f524d52cf71f54eeaa31db8f69f3ce12355ab99b614f955"
Jul 09 18:00:08.655703 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.655682    2573 generic.go:356] "Generic (PLEG): container finished" podID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerID="8317eb20cb66d469fc9369026718457d61fb3489e5af3c88f4535c990cf6dc59" exitCode=0
Jul 09 18:00:08.655834 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.655765    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" event={"ID":"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c","Type":"ContainerDied","Data":"8317eb20cb66d469fc9369026718457d61fb3489e5af3c88f4535c990cf6dc59"}
Jul 09 18:00:08.666335 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.666311    2573 scope.go:117] "RemoveContainer" containerID="5c3c2a35f68eed8a6ffb93f3347140559d1cb696b46e54c58b26f7c1947cce4a"
Jul 09 18:00:08.674403 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.674368    2573 scope.go:117] "RemoveContainer" containerID="ef01c8c71c3b14b5f8b47e463f2c5478542c67f324136cc0f951f49e95b50d65"
Jul 09 18:00:08.677548 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.677484    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"]
Jul 09 18:00:08.679499 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.679470    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9"]
Jul 09 18:00:08.684440 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.684408    2573 scope.go:117] "RemoveContainer" containerID="28429fbf9e6afe082f524d52cf71f54eeaa31db8f69f3ce12355ab99b614f955"
Jul 09 18:00:08.684703 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:00:08.684684    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28429fbf9e6afe082f524d52cf71f54eeaa31db8f69f3ce12355ab99b614f955\": container with ID starting with 28429fbf9e6afe082f524d52cf71f54eeaa31db8f69f3ce12355ab99b614f955 not found: ID does not exist" containerID="28429fbf9e6afe082f524d52cf71f54eeaa31db8f69f3ce12355ab99b614f955"
Jul 09 18:00:08.684807 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.684710    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28429fbf9e6afe082f524d52cf71f54eeaa31db8f69f3ce12355ab99b614f955"} err="failed to get container status \"28429fbf9e6afe082f524d52cf71f54eeaa31db8f69f3ce12355ab99b614f955\": rpc error: code = NotFound desc = could not find container \"28429fbf9e6afe082f524d52cf71f54eeaa31db8f69f3ce12355ab99b614f955\": container with ID starting with 28429fbf9e6afe082f524d52cf71f54eeaa31db8f69f3ce12355ab99b614f955 not found: ID does not exist"
Jul 09 18:00:08.684807 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.684744    2573 scope.go:117] "RemoveContainer" containerID="5c3c2a35f68eed8a6ffb93f3347140559d1cb696b46e54c58b26f7c1947cce4a"
Jul 09 18:00:08.685005 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:00:08.684987    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c3c2a35f68eed8a6ffb93f3347140559d1cb696b46e54c58b26f7c1947cce4a\": container with ID starting with 5c3c2a35f68eed8a6ffb93f3347140559d1cb696b46e54c58b26f7c1947cce4a not found: ID does not exist" containerID="5c3c2a35f68eed8a6ffb93f3347140559d1cb696b46e54c58b26f7c1947cce4a"
Jul 09 18:00:08.685049 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.685012    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c3c2a35f68eed8a6ffb93f3347140559d1cb696b46e54c58b26f7c1947cce4a"} err="failed to get container status \"5c3c2a35f68eed8a6ffb93f3347140559d1cb696b46e54c58b26f7c1947cce4a\": rpc error: code = NotFound desc = could not find container \"5c3c2a35f68eed8a6ffb93f3347140559d1cb696b46e54c58b26f7c1947cce4a\": container with ID starting with 5c3c2a35f68eed8a6ffb93f3347140559d1cb696b46e54c58b26f7c1947cce4a not found: ID does not exist"
Jul 09 18:00:08.685049 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.685027    2573 scope.go:117] "RemoveContainer" containerID="ef01c8c71c3b14b5f8b47e463f2c5478542c67f324136cc0f951f49e95b50d65"
Jul 09 18:00:08.685244 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:00:08.685223    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef01c8c71c3b14b5f8b47e463f2c5478542c67f324136cc0f951f49e95b50d65\": container with ID starting with ef01c8c71c3b14b5f8b47e463f2c5478542c67f324136cc0f951f49e95b50d65 not found: ID does not exist" containerID="ef01c8c71c3b14b5f8b47e463f2c5478542c67f324136cc0f951f49e95b50d65"
Jul 09 18:00:08.685286 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:08.685251    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef01c8c71c3b14b5f8b47e463f2c5478542c67f324136cc0f951f49e95b50d65"} err="failed to get container status \"ef01c8c71c3b14b5f8b47e463f2c5478542c67f324136cc0f951f49e95b50d65\": rpc error: code = NotFound desc = could not find container \"ef01c8c71c3b14b5f8b47e463f2c5478542c67f324136cc0f951f49e95b50d65\": container with ID starting with ef01c8c71c3b14b5f8b47e463f2c5478542c67f324136cc0f951f49e95b50d65 not found: ID does not exist"
Jul 09 18:00:09.663569 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:09.663537    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" event={"ID":"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c","Type":"ContainerStarted","Data":"1486d249218e567b3019a55ec765adab96c21d331c47448ecffb7b327360cf6f"}
Jul 09 18:00:09.663994 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:09.663624    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" event={"ID":"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c","Type":"ContainerStarted","Data":"630bd4642af79722229bf306a6c25b6a86abe93efb818390c2314e5f3eebb0bb"}
Jul 09 18:00:09.663994 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:09.663818    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:09.686137 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:09.686081    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" podStartSLOduration=2.8897319489999997 podStartE2EDuration="6.686065518s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.796333569s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:00:09.685063924 +0000 UTC m=+1253.698593040" watchObservedRunningTime="2026-07-09 18:00:09.686065518 +0000 UTC m=+1253.699594632"
Jul 09 18:00:10.527564 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:10.527526    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" path="/var/lib/kubelet/pods/0eacc765-85e4-4319-88f6-88b545d56ac2/volumes"
Jul 09 18:00:10.667765 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:10.667713    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:10.669176 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:10.669144    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 09 18:00:11.672813 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:11.672772    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 09 18:00:16.678533 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:16.678439    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:00:16.679025 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:16.678994    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 09 18:00:26.679503 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:26.679461    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 09 18:00:36.679564 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:36.679521    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 09 18:00:46.679783 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:46.679738    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 09 18:00:56.679588 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:00:56.679549    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 09 18:01:06.678869 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:06.678830    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 09 18:01:16.678928 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:16.678888    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 09 18:01:26.679651 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:26.679617    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:01:33.974736 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:33.974695    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"]
Jul 09 18:01:33.975147 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:33.975023    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="storage-initializer"
Jul 09 18:01:33.975147 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:33.975037    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="storage-initializer"
Jul 09 18:01:33.975147 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:33.975050    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kube-rbac-proxy"
Jul 09 18:01:33.975147 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:33.975056    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kube-rbac-proxy"
Jul 09 18:01:33.975147 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:33.975072    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container"
Jul 09 18:01:33.975147 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:33.975078    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container"
Jul 09 18:01:33.975147 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:33.975133    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kserve-container"
Jul 09 18:01:33.975147 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:33.975142    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eacc765-85e4-4319-88f6-88b545d56ac2" containerName="kube-rbac-proxy"
Jul 09 18:01:33.978385 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:33.978368    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:33.981617 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:33.981591    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-21e027-predictor-serving-cert\""
Jul 09 18:01:33.981769 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:33.981591    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-21e027\""
Jul 09 18:01:33.981769 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:33.981644    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-21e027-dockercfg-2jlmt\""
Jul 09 18:01:33.982049 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:33.982032    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-21e027-kube-rbac-proxy-sar-config\""
Jul 09 18:01:33.989617 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:33.989594    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"]
Jul 09 18:01:34.058047 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.058012    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3536bec-6d76-4d3e-ae0c-9d38806500f8-proxy-tls\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.058200 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.058054    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kserve-provision-location\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.058200 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.058107    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3536bec-6d76-4d3e-ae0c-9d38806500f8-cabundle-cert\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.058200 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.058143    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.058200 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.058182    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl6cp\" (UniqueName: \"kubernetes.io/projected/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kube-api-access-wl6cp\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.158662 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.158634    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kserve-provision-location\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.158841 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.158672    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3536bec-6d76-4d3e-ae0c-9d38806500f8-cabundle-cert\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.158841 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.158701    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.158841 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.158750    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl6cp\" (UniqueName: \"kubernetes.io/projected/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kube-api-access-wl6cp\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.158841 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.158827    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3536bec-6d76-4d3e-ae0c-9d38806500f8-proxy-tls\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.159001 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:01:34.158929    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-21e027-predictor-serving-cert: secret "isvc-secondary-21e027-predictor-serving-cert" not found
Jul 09 18:01:34.159001 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:01:34.158996    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3536bec-6d76-4d3e-ae0c-9d38806500f8-proxy-tls podName:d3536bec-6d76-4d3e-ae0c-9d38806500f8 nodeName:}" failed. No retries permitted until 2026-07-09 18:01:34.658974773 +0000 UTC m=+1338.672503865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d3536bec-6d76-4d3e-ae0c-9d38806500f8-proxy-tls") pod "isvc-secondary-21e027-predictor-6d849b584c-p4tz8" (UID: "d3536bec-6d76-4d3e-ae0c-9d38806500f8") : secret "isvc-secondary-21e027-predictor-serving-cert" not found
Jul 09 18:01:34.159142 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.159124    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kserve-provision-location\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.159319 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.159302    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3536bec-6d76-4d3e-ae0c-9d38806500f8-cabundle-cert\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.159426 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.159409    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.168625 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.168602    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl6cp\" (UniqueName: \"kubernetes.io/projected/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kube-api-access-wl6cp\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.661963 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.661932    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3536bec-6d76-4d3e-ae0c-9d38806500f8-proxy-tls\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.664297 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.664273    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3536bec-6d76-4d3e-ae0c-9d38806500f8-proxy-tls\") pod \"isvc-secondary-21e027-predictor-6d849b584c-p4tz8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") " pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:34.889357 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:34.889323    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:35.020480 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:35.020454    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"]
Jul 09 18:01:35.022355 ip-10-0-143-142 kubenswrapper[2573]: W0709 18:01:35.022329    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3536bec_6d76_4d3e_ae0c_9d38806500f8.slice/crio-d6431c9d5d419548564511c71f27da6d8825977115adb4411a1f95c2824a712b WatchSource:0}: Error finding container d6431c9d5d419548564511c71f27da6d8825977115adb4411a1f95c2824a712b: Status 404 returned error can't find the container with id d6431c9d5d419548564511c71f27da6d8825977115adb4411a1f95c2824a712b
Jul 09 18:01:36.000943 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:36.000899    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8" event={"ID":"d3536bec-6d76-4d3e-ae0c-9d38806500f8","Type":"ContainerStarted","Data":"2d788813d49999204490178e5f366d18c2c7411d39e01160a14a6095e155e428"}
Jul 09 18:01:36.001146 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:36.000999    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8" event={"ID":"d3536bec-6d76-4d3e-ae0c-9d38806500f8","Type":"ContainerStarted","Data":"d6431c9d5d419548564511c71f27da6d8825977115adb4411a1f95c2824a712b"}
Jul 09 18:01:41.021353 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:41.021324    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-21e027-predictor-6d849b584c-p4tz8_d3536bec-6d76-4d3e-ae0c-9d38806500f8/storage-initializer/0.log"
Jul 09 18:01:41.021765 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:41.021361    2573 generic.go:356] "Generic (PLEG): container finished" podID="d3536bec-6d76-4d3e-ae0c-9d38806500f8" containerID="2d788813d49999204490178e5f366d18c2c7411d39e01160a14a6095e155e428" exitCode=1
Jul 09 18:01:41.021765 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:41.021403    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8" event={"ID":"d3536bec-6d76-4d3e-ae0c-9d38806500f8","Type":"ContainerDied","Data":"2d788813d49999204490178e5f366d18c2c7411d39e01160a14a6095e155e428"}
Jul 09 18:01:42.026810 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:42.026776    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-21e027-predictor-6d849b584c-p4tz8_d3536bec-6d76-4d3e-ae0c-9d38806500f8/storage-initializer/0.log"
Jul 09 18:01:42.027147 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:42.026899    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8" event={"ID":"d3536bec-6d76-4d3e-ae0c-9d38806500f8","Type":"ContainerStarted","Data":"39f2fed66ea47f6db292be62da22af71ae712a8d95e16806e6289bd1d586a003"}
Jul 09 18:01:47.049148 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:47.049117    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-21e027-predictor-6d849b584c-p4tz8_d3536bec-6d76-4d3e-ae0c-9d38806500f8/storage-initializer/1.log"
Jul 09 18:01:47.049542 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:47.049491    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-21e027-predictor-6d849b584c-p4tz8_d3536bec-6d76-4d3e-ae0c-9d38806500f8/storage-initializer/0.log"
Jul 09 18:01:47.049542 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:47.049521    2573 generic.go:356] "Generic (PLEG): container finished" podID="d3536bec-6d76-4d3e-ae0c-9d38806500f8" containerID="39f2fed66ea47f6db292be62da22af71ae712a8d95e16806e6289bd1d586a003" exitCode=1
Jul 09 18:01:47.049639 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:47.049609    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8" event={"ID":"d3536bec-6d76-4d3e-ae0c-9d38806500f8","Type":"ContainerDied","Data":"39f2fed66ea47f6db292be62da22af71ae712a8d95e16806e6289bd1d586a003"}
Jul 09 18:01:47.049708 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:47.049693    2573 scope.go:117] "RemoveContainer" containerID="2d788813d49999204490178e5f366d18c2c7411d39e01160a14a6095e155e428"
Jul 09 18:01:47.050026 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:47.049998    2573 scope.go:117] "RemoveContainer" containerID="2d788813d49999204490178e5f366d18c2c7411d39e01160a14a6095e155e428"
Jul 09 18:01:47.060258 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:01:47.060238    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-21e027-predictor-6d849b584c-p4tz8_kserve-ci-e2e-test(d3536bec-6d76-4d3e-ae0c-9d38806500f8)\"" pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8" podUID="d3536bec-6d76-4d3e-ae0c-9d38806500f8"
Jul 09 18:01:47.060495 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:01:47.060467    2573 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-21e027-predictor-6d849b584c-p4tz8_kserve-ci-e2e-test_d3536bec-6d76-4d3e-ae0c-9d38806500f8_0 in pod sandbox d6431c9d5d419548564511c71f27da6d8825977115adb4411a1f95c2824a712b from index: no such id: '2d788813d49999204490178e5f366d18c2c7411d39e01160a14a6095e155e428'" containerID="2d788813d49999204490178e5f366d18c2c7411d39e01160a14a6095e155e428"
Jul 09 18:01:47.060563 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:47.060503    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d788813d49999204490178e5f366d18c2c7411d39e01160a14a6095e155e428"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-21e027-predictor-6d849b584c-p4tz8_kserve-ci-e2e-test_d3536bec-6d76-4d3e-ae0c-9d38806500f8_0 in pod sandbox d6431c9d5d419548564511c71f27da6d8825977115adb4411a1f95c2824a712b from index: no such id: '2d788813d49999204490178e5f366d18c2c7411d39e01160a14a6095e155e428'"
Jul 09 18:01:48.055322 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:48.055297    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-21e027-predictor-6d849b584c-p4tz8_d3536bec-6d76-4d3e-ae0c-9d38806500f8/storage-initializer/1.log"
Jul 09 18:01:51.995460 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:51.995423    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"]
Jul 09 18:01:51.995885 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:51.995699    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kserve-container" containerID="cri-o://630bd4642af79722229bf306a6c25b6a86abe93efb818390c2314e5f3eebb0bb" gracePeriod=30
Jul 09 18:01:51.995885 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:51.995738    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kube-rbac-proxy" containerID="cri-o://1486d249218e567b3019a55ec765adab96c21d331c47448ecffb7b327360cf6f" gracePeriod=30
Jul 09 18:01:52.050558 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.050528    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"]
Jul 09 18:01:52.195771 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.195750    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-21e027-predictor-6d849b584c-p4tz8_d3536bec-6d76-4d3e-ae0c-9d38806500f8/storage-initializer/1.log"
Jul 09 18:01:52.198055 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.198041    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:52.297781 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.297697    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3536bec-6d76-4d3e-ae0c-9d38806500f8-proxy-tls\") pod \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") "
Jul 09 18:01:52.297920 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.297785    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kserve-provision-location\") pod \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") "
Jul 09 18:01:52.297920 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.297809    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl6cp\" (UniqueName: \"kubernetes.io/projected/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kube-api-access-wl6cp\") pod \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") "
Jul 09 18:01:52.297920 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.297830    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kube-rbac-proxy-sar-config\") pod \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") "
Jul 09 18:01:52.297920 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.297853    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3536bec-6d76-4d3e-ae0c-9d38806500f8-cabundle-cert\") pod \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\" (UID: \"d3536bec-6d76-4d3e-ae0c-9d38806500f8\") "
Jul 09 18:01:52.298147 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.298080    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3536bec-6d76-4d3e-ae0c-9d38806500f8" (UID: "d3536bec-6d76-4d3e-ae0c-9d38806500f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:01:52.298204 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.298183    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d3536bec-6d76-4d3e-ae0c-9d38806500f8" (UID: "d3536bec-6d76-4d3e-ae0c-9d38806500f8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:01:52.298253 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.298234    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3536bec-6d76-4d3e-ae0c-9d38806500f8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3536bec-6d76-4d3e-ae0c-9d38806500f8" (UID: "d3536bec-6d76-4d3e-ae0c-9d38806500f8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:01:52.299863 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.299837    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kube-api-access-wl6cp" (OuterVolumeSpecName: "kube-api-access-wl6cp") pod "d3536bec-6d76-4d3e-ae0c-9d38806500f8" (UID: "d3536bec-6d76-4d3e-ae0c-9d38806500f8"). InnerVolumeSpecName "kube-api-access-wl6cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:01:52.299863 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.299842    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3536bec-6d76-4d3e-ae0c-9d38806500f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3536bec-6d76-4d3e-ae0c-9d38806500f8" (UID: "d3536bec-6d76-4d3e-ae0c-9d38806500f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:01:52.398503 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.398476    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3536bec-6d76-4d3e-ae0c-9d38806500f8-cabundle-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:01:52.398503 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.398498    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3536bec-6d76-4d3e-ae0c-9d38806500f8-proxy-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:01:52.398653 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.398508    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kserve-provision-location\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:01:52.398653 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.398517    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl6cp\" (UniqueName: \"kubernetes.io/projected/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kube-api-access-wl6cp\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:01:52.398653 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:52.398527    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3536bec-6d76-4d3e-ae0c-9d38806500f8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:01:53.079428 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:53.079392    2573 generic.go:356] "Generic (PLEG): container finished" podID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerID="1486d249218e567b3019a55ec765adab96c21d331c47448ecffb7b327360cf6f" exitCode=2
Jul 09 18:01:53.079856 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:53.079456    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" event={"ID":"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c","Type":"ContainerDied","Data":"1486d249218e567b3019a55ec765adab96c21d331c47448ecffb7b327360cf6f"}
Jul 09 18:01:53.080851 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:53.080833    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-21e027-predictor-6d849b584c-p4tz8_d3536bec-6d76-4d3e-ae0c-9d38806500f8/storage-initializer/1.log"
Jul 09 18:01:53.080987 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:53.080897    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8" event={"ID":"d3536bec-6d76-4d3e-ae0c-9d38806500f8","Type":"ContainerDied","Data":"d6431c9d5d419548564511c71f27da6d8825977115adb4411a1f95c2824a712b"}
Jul 09 18:01:53.080987 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:53.080934    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"
Jul 09 18:01:53.081102 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:53.080939    2573 scope.go:117] "RemoveContainer" containerID="39f2fed66ea47f6db292be62da22af71ae712a8d95e16806e6289bd1d586a003"
Jul 09 18:01:53.125772 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:53.125738    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"]
Jul 09 18:01:53.128127 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:53.128099    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8"]
Jul 09 18:01:54.527043 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:54.527008    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3536bec-6d76-4d3e-ae0c-9d38806500f8" path="/var/lib/kubelet/pods/d3536bec-6d76-4d3e-ae0c-9d38806500f8/volumes"
Jul 09 18:01:56.535631 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.535608    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:01:56.634534 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.634457    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-proxy-tls\") pod \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") "
Jul 09 18:01:56.634534 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.634511    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kube-rbac-proxy-sar-config\") pod \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") "
Jul 09 18:01:56.634777 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.634540    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-cabundle-cert\") pod \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") "
Jul 09 18:01:56.634777 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.634576    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6wlv\" (UniqueName: \"kubernetes.io/projected/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kube-api-access-l6wlv\") pod \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") "
Jul 09 18:01:56.634777 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.634613    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kserve-provision-location\") pod \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\" (UID: \"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c\") "
Jul 09 18:01:56.635279 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.634973    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" (UID: "9e5eaa6d-620e-4d88-b34f-a1936c79fe1c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:01:56.635279 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.634983    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" (UID: "9e5eaa6d-620e-4d88-b34f-a1936c79fe1c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:01:56.635279 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.635004    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" (UID: "9e5eaa6d-620e-4d88-b34f-a1936c79fe1c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:01:56.636709 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.636688    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" (UID: "9e5eaa6d-620e-4d88-b34f-a1936c79fe1c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:01:56.636877 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.636855    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kube-api-access-l6wlv" (OuterVolumeSpecName: "kube-api-access-l6wlv") pod "9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" (UID: "9e5eaa6d-620e-4d88-b34f-a1936c79fe1c"). InnerVolumeSpecName "kube-api-access-l6wlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:01:56.735855 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.735831    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-proxy-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:01:56.735855 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.735852    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:01:56.735998 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.735862    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-cabundle-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:01:56.735998 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.735872    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6wlv\" (UniqueName: \"kubernetes.io/projected/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kube-api-access-l6wlv\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:01:56.735998 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:56.735880    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c-kserve-provision-location\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:01:57.103085 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.103053    2573 generic.go:356] "Generic (PLEG): container finished" podID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerID="630bd4642af79722229bf306a6c25b6a86abe93efb818390c2314e5f3eebb0bb" exitCode=0
Jul 09 18:01:57.103292 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.103123    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"
Jul 09 18:01:57.103292 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.103139    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" event={"ID":"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c","Type":"ContainerDied","Data":"630bd4642af79722229bf306a6c25b6a86abe93efb818390c2314e5f3eebb0bb"}
Jul 09 18:01:57.103292 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.103199    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5" event={"ID":"9e5eaa6d-620e-4d88-b34f-a1936c79fe1c","Type":"ContainerDied","Data":"5943b8b2a6295bb310ea56ea46afc4c529dd11ff00ffa7471edaa64b3103f4dd"}
Jul 09 18:01:57.103292 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.103226    2573 scope.go:117] "RemoveContainer" containerID="1486d249218e567b3019a55ec765adab96c21d331c47448ecffb7b327360cf6f"
Jul 09 18:01:57.111384 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.111368    2573 scope.go:117] "RemoveContainer" containerID="630bd4642af79722229bf306a6c25b6a86abe93efb818390c2314e5f3eebb0bb"
Jul 09 18:01:57.118713 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.118697    2573 scope.go:117] "RemoveContainer" containerID="8317eb20cb66d469fc9369026718457d61fb3489e5af3c88f4535c990cf6dc59"
Jul 09 18:01:57.125545 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.125525    2573 scope.go:117] "RemoveContainer" containerID="1486d249218e567b3019a55ec765adab96c21d331c47448ecffb7b327360cf6f"
Jul 09 18:01:57.125814 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:01:57.125790    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1486d249218e567b3019a55ec765adab96c21d331c47448ecffb7b327360cf6f\": container with ID starting with 1486d249218e567b3019a55ec765adab96c21d331c47448ecffb7b327360cf6f not found: ID does not exist" containerID="1486d249218e567b3019a55ec765adab96c21d331c47448ecffb7b327360cf6f"
Jul 09 18:01:57.125871 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.125814    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1486d249218e567b3019a55ec765adab96c21d331c47448ecffb7b327360cf6f"} err="failed to get container status \"1486d249218e567b3019a55ec765adab96c21d331c47448ecffb7b327360cf6f\": rpc error: code = NotFound desc = could not find container \"1486d249218e567b3019a55ec765adab96c21d331c47448ecffb7b327360cf6f\": container with ID starting with 1486d249218e567b3019a55ec765adab96c21d331c47448ecffb7b327360cf6f not found: ID does not exist"
Jul 09 18:01:57.125871 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.125832    2573 scope.go:117] "RemoveContainer" containerID="630bd4642af79722229bf306a6c25b6a86abe93efb818390c2314e5f3eebb0bb"
Jul 09 18:01:57.126058 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:01:57.126039    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630bd4642af79722229bf306a6c25b6a86abe93efb818390c2314e5f3eebb0bb\": container with ID starting with 630bd4642af79722229bf306a6c25b6a86abe93efb818390c2314e5f3eebb0bb not found: ID does not exist" containerID="630bd4642af79722229bf306a6c25b6a86abe93efb818390c2314e5f3eebb0bb"
Jul 09 18:01:57.126129 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.126067    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630bd4642af79722229bf306a6c25b6a86abe93efb818390c2314e5f3eebb0bb"} err="failed to get container status \"630bd4642af79722229bf306a6c25b6a86abe93efb818390c2314e5f3eebb0bb\": rpc error: code = NotFound desc = could not find container \"630bd4642af79722229bf306a6c25b6a86abe93efb818390c2314e5f3eebb0bb\": container with ID starting with 630bd4642af79722229bf306a6c25b6a86abe93efb818390c2314e5f3eebb0bb not found: ID does not exist"
Jul 09 18:01:57.126129 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.126089    2573 scope.go:117] "RemoveContainer" containerID="8317eb20cb66d469fc9369026718457d61fb3489e5af3c88f4535c990cf6dc59"
Jul 09 18:01:57.126333 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:01:57.126316    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8317eb20cb66d469fc9369026718457d61fb3489e5af3c88f4535c990cf6dc59\": container with ID starting with 8317eb20cb66d469fc9369026718457d61fb3489e5af3c88f4535c990cf6dc59 not found: ID does not exist" containerID="8317eb20cb66d469fc9369026718457d61fb3489e5af3c88f4535c990cf6dc59"
Jul 09 18:01:57.126374 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.126338    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8317eb20cb66d469fc9369026718457d61fb3489e5af3c88f4535c990cf6dc59"} err="failed to get container status \"8317eb20cb66d469fc9369026718457d61fb3489e5af3c88f4535c990cf6dc59\": rpc error: code = NotFound desc = could not find container \"8317eb20cb66d469fc9369026718457d61fb3489e5af3c88f4535c990cf6dc59\": container with ID starting with 8317eb20cb66d469fc9369026718457d61fb3489e5af3c88f4535c990cf6dc59 not found: ID does not exist"
Jul 09 18:01:57.130484 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.130463    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"]
Jul 09 18:01:57.136812 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.136791    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5"]
Jul 09 18:01:57.265822 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.265789    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"]
Jul 09 18:01:57.266066 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.266054    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kserve-container"
Jul 09 18:01:57.266104 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.266067    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kserve-container"
Jul 09 18:01:57.266104 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.266077    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3536bec-6d76-4d3e-ae0c-9d38806500f8" containerName="storage-initializer"
Jul 09 18:01:57.266104 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.266082    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3536bec-6d76-4d3e-ae0c-9d38806500f8" containerName="storage-initializer"
Jul 09 18:01:57.266104 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.266091    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="storage-initializer"
Jul 09 18:01:57.266104 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.266096    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="storage-initializer"
Jul 09 18:01:57.266258 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.266110    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3536bec-6d76-4d3e-ae0c-9d38806500f8" containerName="storage-initializer"
Jul 09 18:01:57.266258 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.266115    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3536bec-6d76-4d3e-ae0c-9d38806500f8" containerName="storage-initializer"
Jul 09 18:01:57.266258 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.266123    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kube-rbac-proxy"
Jul 09 18:01:57.266258 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.266129    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kube-rbac-proxy"
Jul 09 18:01:57.266258 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.266171    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3536bec-6d76-4d3e-ae0c-9d38806500f8" containerName="storage-initializer"
Jul 09 18:01:57.266258 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.266178    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kserve-container"
Jul 09 18:01:57.266258 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.266187    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3536bec-6d76-4d3e-ae0c-9d38806500f8" containerName="storage-initializer"
Jul 09 18:01:57.266258 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.266193    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" containerName="kube-rbac-proxy"
Jul 09 18:01:57.270456 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.270437    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.273882 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.273860    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:01:57.273987 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.273866    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-afe32a-predictor-serving-cert\""
Jul 09 18:01:57.274046 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.274025    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:01:57.274422 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.274407    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:01:57.274486 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.274417    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-afe32a-kube-rbac-proxy-sar-config\""
Jul 09 18:01:57.274486 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.274478    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-afe32a\""
Jul 09 18:01:57.274645 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.274633    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-afe32a-dockercfg-xlkrq\""
Jul 09 18:01:57.288435 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.288414    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"]
Jul 09 18:01:57.340701 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.340674    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00df6b45-2c52-4b7e-95ee-058f6e578fae-proxy-tls\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.340850 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.340740    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd96k\" (UniqueName: \"kubernetes.io/projected/00df6b45-2c52-4b7e-95ee-058f6e578fae-kube-api-access-sd96k\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.340850 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.340776    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00df6b45-2c52-4b7e-95ee-058f6e578fae-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.340850 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.340803    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00df6b45-2c52-4b7e-95ee-058f6e578fae-kserve-provision-location\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.340990 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.340861    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00df6b45-2c52-4b7e-95ee-058f6e578fae-cabundle-cert\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.442025 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.441951    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00df6b45-2c52-4b7e-95ee-058f6e578fae-kserve-provision-location\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.442025 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.441991    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00df6b45-2c52-4b7e-95ee-058f6e578fae-cabundle-cert\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.442025 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.442013    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00df6b45-2c52-4b7e-95ee-058f6e578fae-proxy-tls\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.442252 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:01:57.442110    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-serving-cert: secret "isvc-init-fail-afe32a-predictor-serving-cert" not found
Jul 09 18:01:57.442252 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.442121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd96k\" (UniqueName: \"kubernetes.io/projected/00df6b45-2c52-4b7e-95ee-058f6e578fae-kube-api-access-sd96k\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.442252 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:01:57.442163    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00df6b45-2c52-4b7e-95ee-058f6e578fae-proxy-tls podName:00df6b45-2c52-4b7e-95ee-058f6e578fae nodeName:}" failed. No retries permitted until 2026-07-09 18:01:57.942147597 +0000 UTC m=+1361.955676688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/00df6b45-2c52-4b7e-95ee-058f6e578fae-proxy-tls") pod "isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj" (UID: "00df6b45-2c52-4b7e-95ee-058f6e578fae") : secret "isvc-init-fail-afe32a-predictor-serving-cert" not found
Jul 09 18:01:57.442252 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.442193    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00df6b45-2c52-4b7e-95ee-058f6e578fae-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.442467 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.442334    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00df6b45-2c52-4b7e-95ee-058f6e578fae-kserve-provision-location\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.442735 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.442697    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00df6b45-2c52-4b7e-95ee-058f6e578fae-cabundle-cert\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.442818 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.442699    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00df6b45-2c52-4b7e-95ee-058f6e578fae-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.454688 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.454662    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd96k\" (UniqueName: \"kubernetes.io/projected/00df6b45-2c52-4b7e-95ee-058f6e578fae-kube-api-access-sd96k\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.947598 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.947564    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00df6b45-2c52-4b7e-95ee-058f6e578fae-proxy-tls\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:57.950148 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:57.950121    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00df6b45-2c52-4b7e-95ee-058f6e578fae-proxy-tls\") pod \"isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") " pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:58.179990 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:58.179954    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:01:58.317549 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:58.317490    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"]
Jul 09 18:01:58.319647 ip-10-0-143-142 kubenswrapper[2573]: W0709 18:01:58.319618    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00df6b45_2c52_4b7e_95ee_058f6e578fae.slice/crio-f432223e8532dbf9c45cb53c1775befa2f37678ecbce30571b02e91889b26dbd WatchSource:0}: Error finding container f432223e8532dbf9c45cb53c1775befa2f37678ecbce30571b02e91889b26dbd: Status 404 returned error can't find the container with id f432223e8532dbf9c45cb53c1775befa2f37678ecbce30571b02e91889b26dbd
Jul 09 18:01:58.528083 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:58.528048    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e5eaa6d-620e-4d88-b34f-a1936c79fe1c" path="/var/lib/kubelet/pods/9e5eaa6d-620e-4d88-b34f-a1936c79fe1c/volumes"
Jul 09 18:01:59.113487 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:59.113449    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj" event={"ID":"00df6b45-2c52-4b7e-95ee-058f6e578fae","Type":"ContainerStarted","Data":"9cff7745be1c0e67f126d087b304dc52d1ffef5f1e59d7c2a6d43d1e33d405f7"}
Jul 09 18:01:59.113894 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:01:59.113538    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj" event={"ID":"00df6b45-2c52-4b7e-95ee-058f6e578fae","Type":"ContainerStarted","Data":"f432223e8532dbf9c45cb53c1775befa2f37678ecbce30571b02e91889b26dbd"}
Jul 09 18:02:02.126215 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:02.126186    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj_00df6b45-2c52-4b7e-95ee-058f6e578fae/storage-initializer/0.log"
Jul 09 18:02:02.126580 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:02.126224    2573 generic.go:356] "Generic (PLEG): container finished" podID="00df6b45-2c52-4b7e-95ee-058f6e578fae" containerID="9cff7745be1c0e67f126d087b304dc52d1ffef5f1e59d7c2a6d43d1e33d405f7" exitCode=1
Jul 09 18:02:02.126580 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:02.126307    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj" event={"ID":"00df6b45-2c52-4b7e-95ee-058f6e578fae","Type":"ContainerDied","Data":"9cff7745be1c0e67f126d087b304dc52d1ffef5f1e59d7c2a6d43d1e33d405f7"}
Jul 09 18:02:03.132006 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:03.131975    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj_00df6b45-2c52-4b7e-95ee-058f6e578fae/storage-initializer/0.log"
Jul 09 18:02:03.132387 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:03.132079    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj" event={"ID":"00df6b45-2c52-4b7e-95ee-058f6e578fae","Type":"ContainerStarted","Data":"feca97e99009da1c55564caa97161583e00a37022dd89e9be0ed7e772d3dcf35"}
Jul 09 18:02:07.270841 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:07.270801    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"]
Jul 09 18:02:07.271231 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:07.271005    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj" podUID="00df6b45-2c52-4b7e-95ee-058f6e578fae" containerName="storage-initializer" containerID="cri-o://feca97e99009da1c55564caa97161583e00a37022dd89e9be0ed7e772d3dcf35" gracePeriod=30
Jul 09 18:02:07.821228 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:07.821205    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj_00df6b45-2c52-4b7e-95ee-058f6e578fae/storage-initializer/1.log"
Jul 09 18:02:07.821600 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:07.821586    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj_00df6b45-2c52-4b7e-95ee-058f6e578fae/storage-initializer/0.log"
Jul 09 18:02:07.823857 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:07.823838    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:02:07.932511 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:07.932406    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00df6b45-2c52-4b7e-95ee-058f6e578fae-kube-rbac-proxy-sar-config\") pod \"00df6b45-2c52-4b7e-95ee-058f6e578fae\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") "
Jul 09 18:02:07.932511 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:07.932479    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00df6b45-2c52-4b7e-95ee-058f6e578fae-kserve-provision-location\") pod \"00df6b45-2c52-4b7e-95ee-058f6e578fae\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") "
Jul 09 18:02:07.932511 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:07.932504    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd96k\" (UniqueName: \"kubernetes.io/projected/00df6b45-2c52-4b7e-95ee-058f6e578fae-kube-api-access-sd96k\") pod \"00df6b45-2c52-4b7e-95ee-058f6e578fae\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") "
Jul 09 18:02:07.932830 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:07.932539    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00df6b45-2c52-4b7e-95ee-058f6e578fae-cabundle-cert\") pod \"00df6b45-2c52-4b7e-95ee-058f6e578fae\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") "
Jul 09 18:02:07.932830 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:07.932558    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00df6b45-2c52-4b7e-95ee-058f6e578fae-proxy-tls\") pod \"00df6b45-2c52-4b7e-95ee-058f6e578fae\" (UID: \"00df6b45-2c52-4b7e-95ee-058f6e578fae\") "
Jul 09 18:02:07.932830 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:07.932761    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00df6b45-2c52-4b7e-95ee-058f6e578fae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "00df6b45-2c52-4b7e-95ee-058f6e578fae" (UID: "00df6b45-2c52-4b7e-95ee-058f6e578fae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:02:07.932974 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:07.932900    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00df6b45-2c52-4b7e-95ee-058f6e578fae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "00df6b45-2c52-4b7e-95ee-058f6e578fae" (UID: "00df6b45-2c52-4b7e-95ee-058f6e578fae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:02:07.932974 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:07.932928    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00df6b45-2c52-4b7e-95ee-058f6e578fae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "00df6b45-2c52-4b7e-95ee-058f6e578fae" (UID: "00df6b45-2c52-4b7e-95ee-058f6e578fae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:02:07.934604 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:07.934580    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00df6b45-2c52-4b7e-95ee-058f6e578fae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00df6b45-2c52-4b7e-95ee-058f6e578fae" (UID: "00df6b45-2c52-4b7e-95ee-058f6e578fae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:02:07.934683 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:07.934640    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00df6b45-2c52-4b7e-95ee-058f6e578fae-kube-api-access-sd96k" (OuterVolumeSpecName: "kube-api-access-sd96k") pod "00df6b45-2c52-4b7e-95ee-058f6e578fae" (UID: "00df6b45-2c52-4b7e-95ee-058f6e578fae"). InnerVolumeSpecName "kube-api-access-sd96k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:02:08.033252 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.033204    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00df6b45-2c52-4b7e-95ee-058f6e578fae-cabundle-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:02:08.033252 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.033245    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00df6b45-2c52-4b7e-95ee-058f6e578fae-proxy-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:02:08.033252 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.033255    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00df6b45-2c52-4b7e-95ee-058f6e578fae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:02:08.033252 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.033264    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00df6b45-2c52-4b7e-95ee-058f6e578fae-kserve-provision-location\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:02:08.033507 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.033273    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd96k\" (UniqueName: \"kubernetes.io/projected/00df6b45-2c52-4b7e-95ee-058f6e578fae-kube-api-access-sd96k\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:02:08.153212 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.153184    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj_00df6b45-2c52-4b7e-95ee-058f6e578fae/storage-initializer/1.log"
Jul 09 18:02:08.153563 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.153544    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj_00df6b45-2c52-4b7e-95ee-058f6e578fae/storage-initializer/0.log"
Jul 09 18:02:08.153652 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.153584    2573 generic.go:356] "Generic (PLEG): container finished" podID="00df6b45-2c52-4b7e-95ee-058f6e578fae" containerID="feca97e99009da1c55564caa97161583e00a37022dd89e9be0ed7e772d3dcf35" exitCode=1
Jul 09 18:02:08.153652 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.153632    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"
Jul 09 18:02:08.153774 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.153668    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj" event={"ID":"00df6b45-2c52-4b7e-95ee-058f6e578fae","Type":"ContainerDied","Data":"feca97e99009da1c55564caa97161583e00a37022dd89e9be0ed7e772d3dcf35"}
Jul 09 18:02:08.153774 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.153757    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj" event={"ID":"00df6b45-2c52-4b7e-95ee-058f6e578fae","Type":"ContainerDied","Data":"f432223e8532dbf9c45cb53c1775befa2f37678ecbce30571b02e91889b26dbd"}
Jul 09 18:02:08.153851 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.153785    2573 scope.go:117] "RemoveContainer" containerID="feca97e99009da1c55564caa97161583e00a37022dd89e9be0ed7e772d3dcf35"
Jul 09 18:02:08.162004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.161986    2573 scope.go:117] "RemoveContainer" containerID="9cff7745be1c0e67f126d087b304dc52d1ffef5f1e59d7c2a6d43d1e33d405f7"
Jul 09 18:02:08.169060 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.169043    2573 scope.go:117] "RemoveContainer" containerID="feca97e99009da1c55564caa97161583e00a37022dd89e9be0ed7e772d3dcf35"
Jul 09 18:02:08.169309 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:02:08.169288    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feca97e99009da1c55564caa97161583e00a37022dd89e9be0ed7e772d3dcf35\": container with ID starting with feca97e99009da1c55564caa97161583e00a37022dd89e9be0ed7e772d3dcf35 not found: ID does not exist" containerID="feca97e99009da1c55564caa97161583e00a37022dd89e9be0ed7e772d3dcf35"
Jul 09 18:02:08.169375 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.169322    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feca97e99009da1c55564caa97161583e00a37022dd89e9be0ed7e772d3dcf35"} err="failed to get container status \"feca97e99009da1c55564caa97161583e00a37022dd89e9be0ed7e772d3dcf35\": rpc error: code = NotFound desc = could not find container \"feca97e99009da1c55564caa97161583e00a37022dd89e9be0ed7e772d3dcf35\": container with ID starting with feca97e99009da1c55564caa97161583e00a37022dd89e9be0ed7e772d3dcf35 not found: ID does not exist"
Jul 09 18:02:08.169375 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.169348    2573 scope.go:117] "RemoveContainer" containerID="9cff7745be1c0e67f126d087b304dc52d1ffef5f1e59d7c2a6d43d1e33d405f7"
Jul 09 18:02:08.169605 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:02:08.169589    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cff7745be1c0e67f126d087b304dc52d1ffef5f1e59d7c2a6d43d1e33d405f7\": container with ID starting with 9cff7745be1c0e67f126d087b304dc52d1ffef5f1e59d7c2a6d43d1e33d405f7 not found: ID does not exist" containerID="9cff7745be1c0e67f126d087b304dc52d1ffef5f1e59d7c2a6d43d1e33d405f7"
Jul 09 18:02:08.169648 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.169611    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cff7745be1c0e67f126d087b304dc52d1ffef5f1e59d7c2a6d43d1e33d405f7"} err="failed to get container status \"9cff7745be1c0e67f126d087b304dc52d1ffef5f1e59d7c2a6d43d1e33d405f7\": rpc error: code = NotFound desc = could not find container \"9cff7745be1c0e67f126d087b304dc52d1ffef5f1e59d7c2a6d43d1e33d405f7\": container with ID starting with 9cff7745be1c0e67f126d087b304dc52d1ffef5f1e59d7c2a6d43d1e33d405f7 not found: ID does not exist"
Jul 09 18:02:08.192589 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.192511    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"]
Jul 09 18:02:08.197655 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.197628    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj"]
Jul 09 18:02:08.527612 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:08.527582    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00df6b45-2c52-4b7e-95ee-058f6e578fae" path="/var/lib/kubelet/pods/00df6b45-2c52-4b7e-95ee-058f6e578fae/volumes"
Jul 09 18:02:12.490647 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.490605    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"]
Jul 09 18:02:12.491136 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.490920    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00df6b45-2c52-4b7e-95ee-058f6e578fae" containerName="storage-initializer"
Jul 09 18:02:12.491136 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.490933    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="00df6b45-2c52-4b7e-95ee-058f6e578fae" containerName="storage-initializer"
Jul 09 18:02:12.491136 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.490943    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00df6b45-2c52-4b7e-95ee-058f6e578fae" containerName="storage-initializer"
Jul 09 18:02:12.491136 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.490948    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="00df6b45-2c52-4b7e-95ee-058f6e578fae" containerName="storage-initializer"
Jul 09 18:02:12.491136 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.491002    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="00df6b45-2c52-4b7e-95ee-058f6e578fae" containerName="storage-initializer"
Jul 09 18:02:12.491136 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.491134    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="00df6b45-2c52-4b7e-95ee-058f6e578fae" containerName="storage-initializer"
Jul 09 18:02:12.495585 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.495565    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.498851 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.498825    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-f062e-predictor-serving-cert\""
Jul 09 18:02:12.498987 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.498855    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:02:12.498987 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.498864    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-f062e-kube-rbac-proxy-sar-config\""
Jul 09 18:02:12.498987 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.498907    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:02:12.500354 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.500336    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cfv7h\""
Jul 09 18:02:12.500465 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.500338    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:02:12.500465 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.500368    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:02:12.505385 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.505362    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"]
Jul 09 18:02:12.575872 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.575844    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87ec386b-a148-4878-8742-6fdb583f9173-kserve-provision-location\") pod \"raw-sklearn-f062e-predictor-797bb4b66f-2ws4r\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") " pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.575872 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.575876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87ec386b-a148-4878-8742-6fdb583f9173-cabundle-cert\") pod \"raw-sklearn-f062e-predictor-797bb4b66f-2ws4r\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") " pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.576048 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.575902    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvfdf\" (UniqueName: \"kubernetes.io/projected/87ec386b-a148-4878-8742-6fdb583f9173-kube-api-access-xvfdf\") pod \"raw-sklearn-f062e-predictor-797bb4b66f-2ws4r\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") " pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.576048 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.575975    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87ec386b-a148-4878-8742-6fdb583f9173-proxy-tls\") pod \"raw-sklearn-f062e-predictor-797bb4b66f-2ws4r\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") " pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.576048 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.576029    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87ec386b-a148-4878-8742-6fdb583f9173-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-f062e-predictor-797bb4b66f-2ws4r\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") " pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.677114 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.677085    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvfdf\" (UniqueName: \"kubernetes.io/projected/87ec386b-a148-4878-8742-6fdb583f9173-kube-api-access-xvfdf\") pod \"raw-sklearn-f062e-predictor-797bb4b66f-2ws4r\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") " pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.677256 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.677129    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87ec386b-a148-4878-8742-6fdb583f9173-proxy-tls\") pod \"raw-sklearn-f062e-predictor-797bb4b66f-2ws4r\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") " pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.677256 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.677180    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87ec386b-a148-4878-8742-6fdb583f9173-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-f062e-predictor-797bb4b66f-2ws4r\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") " pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.677256 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.677206    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87ec386b-a148-4878-8742-6fdb583f9173-kserve-provision-location\") pod \"raw-sklearn-f062e-predictor-797bb4b66f-2ws4r\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") " pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.677256 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.677230    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87ec386b-a148-4878-8742-6fdb583f9173-cabundle-cert\") pod \"raw-sklearn-f062e-predictor-797bb4b66f-2ws4r\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") " pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.677597 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.677576    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87ec386b-a148-4878-8742-6fdb583f9173-kserve-provision-location\") pod \"raw-sklearn-f062e-predictor-797bb4b66f-2ws4r\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") " pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.677967 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.677951    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87ec386b-a148-4878-8742-6fdb583f9173-cabundle-cert\") pod \"raw-sklearn-f062e-predictor-797bb4b66f-2ws4r\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") " pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.678030 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.677951    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87ec386b-a148-4878-8742-6fdb583f9173-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-f062e-predictor-797bb4b66f-2ws4r\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") " pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.679864 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.679844    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87ec386b-a148-4878-8742-6fdb583f9173-proxy-tls\") pod \"raw-sklearn-f062e-predictor-797bb4b66f-2ws4r\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") " pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.686173 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.686155    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvfdf\" (UniqueName: \"kubernetes.io/projected/87ec386b-a148-4878-8742-6fdb583f9173-kube-api-access-xvfdf\") pod \"raw-sklearn-f062e-predictor-797bb4b66f-2ws4r\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") " pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.807021 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.806957    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:12.935825 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:12.935800    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"]
Jul 09 18:02:12.937734 ip-10-0-143-142 kubenswrapper[2573]: W0709 18:02:12.937696    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ec386b_a148_4878_8742_6fdb583f9173.slice/crio-c96cdf68ab729c0f6e837e550d28d657e32af604f0bc772a4d9bb1aa35fd2cd1 WatchSource:0}: Error finding container c96cdf68ab729c0f6e837e550d28d657e32af604f0bc772a4d9bb1aa35fd2cd1: Status 404 returned error can't find the container with id c96cdf68ab729c0f6e837e550d28d657e32af604f0bc772a4d9bb1aa35fd2cd1
Jul 09 18:02:13.176824 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:13.176731    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" event={"ID":"87ec386b-a148-4878-8742-6fdb583f9173","Type":"ContainerStarted","Data":"006d0e30951163e483ebb145bb978e8fe7bfbba3534538afb342cca8f38b0cf3"}
Jul 09 18:02:13.176824 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:13.176791    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" event={"ID":"87ec386b-a148-4878-8742-6fdb583f9173","Type":"ContainerStarted","Data":"c96cdf68ab729c0f6e837e550d28d657e32af604f0bc772a4d9bb1aa35fd2cd1"}
Jul 09 18:02:17.192797 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:17.192760    2573 generic.go:356] "Generic (PLEG): container finished" podID="87ec386b-a148-4878-8742-6fdb583f9173" containerID="006d0e30951163e483ebb145bb978e8fe7bfbba3534538afb342cca8f38b0cf3" exitCode=0
Jul 09 18:02:17.193179 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:17.192836    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" event={"ID":"87ec386b-a148-4878-8742-6fdb583f9173","Type":"ContainerDied","Data":"006d0e30951163e483ebb145bb978e8fe7bfbba3534538afb342cca8f38b0cf3"}
Jul 09 18:02:18.198996 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:18.198965    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" event={"ID":"87ec386b-a148-4878-8742-6fdb583f9173","Type":"ContainerStarted","Data":"b26c00e139b5c9ec275d09d9cde654e8cd8238e93df57b0317306c69fea92c29"}
Jul 09 18:02:18.199408 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:18.199025    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" event={"ID":"87ec386b-a148-4878-8742-6fdb583f9173","Type":"ContainerStarted","Data":"21f035f4c3f5c9d9df99c0a8cec1a9f7c03878acd961526611386a4ec3ede64d"}
Jul 09 18:02:18.199408 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:18.199303    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:18.199502 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:18.199454    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:18.200533 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:18.200509    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 09 18:02:18.222874 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:18.222824    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" podStartSLOduration=2.639188208 podStartE2EDuration="6.222801463s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.583613255s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:02:18.222215217 +0000 UTC m=+1382.235744376" watchObservedRunningTime="2026-07-09 18:02:18.222801463 +0000 UTC m=+1382.236330577"
Jul 09 18:02:19.202631 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:19.202595    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 09 18:02:24.207796 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:24.207761    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:02:24.208276 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:24.208252    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 09 18:02:34.208269 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:34.208218    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 09 18:02:44.208474 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:44.208432    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 09 18:02:54.208690 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:02:54.208645    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 09 18:03:04.209032 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:04.208993    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 09 18:03:14.208295 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:14.208202    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 09 18:03:24.208392 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:24.208324    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 09 18:03:34.209502 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:34.209468    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:03:42.640964 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:42.640931    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"]
Jul 09 18:03:42.641436 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:42.641186    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kserve-container" containerID="cri-o://21f035f4c3f5c9d9df99c0a8cec1a9f7c03878acd961526611386a4ec3ede64d" gracePeriod=30
Jul 09 18:03:42.641436 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:42.641256    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kube-rbac-proxy" containerID="cri-o://b26c00e139b5c9ec275d09d9cde654e8cd8238e93df57b0317306c69fea92c29" gracePeriod=30
Jul 09 18:03:43.535588 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:43.535555    2573 generic.go:356] "Generic (PLEG): container finished" podID="87ec386b-a148-4878-8742-6fdb583f9173" containerID="b26c00e139b5c9ec275d09d9cde654e8cd8238e93df57b0317306c69fea92c29" exitCode=2
Jul 09 18:03:43.535770 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:43.535602    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" event={"ID":"87ec386b-a148-4878-8742-6fdb583f9173","Type":"ContainerDied","Data":"b26c00e139b5c9ec275d09d9cde654e8cd8238e93df57b0317306c69fea92c29"}
Jul 09 18:03:44.203061 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:44.203019    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 09 18:03:44.208477 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:44.208435    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 09 18:03:47.177226 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.177203    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:03:47.212226 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.212200    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87ec386b-a148-4878-8742-6fdb583f9173-kserve-provision-location\") pod \"87ec386b-a148-4878-8742-6fdb583f9173\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") "
Jul 09 18:03:47.212361 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.212265    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvfdf\" (UniqueName: \"kubernetes.io/projected/87ec386b-a148-4878-8742-6fdb583f9173-kube-api-access-xvfdf\") pod \"87ec386b-a148-4878-8742-6fdb583f9173\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") "
Jul 09 18:03:47.212361 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.212304    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87ec386b-a148-4878-8742-6fdb583f9173-kube-rbac-proxy-sar-config\") pod \"87ec386b-a148-4878-8742-6fdb583f9173\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") "
Jul 09 18:03:47.212361 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.212323    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87ec386b-a148-4878-8742-6fdb583f9173-cabundle-cert\") pod \"87ec386b-a148-4878-8742-6fdb583f9173\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") "
Jul 09 18:03:47.212514 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.212368    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87ec386b-a148-4878-8742-6fdb583f9173-proxy-tls\") pod \"87ec386b-a148-4878-8742-6fdb583f9173\" (UID: \"87ec386b-a148-4878-8742-6fdb583f9173\") "
Jul 09 18:03:47.212571 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.212540    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ec386b-a148-4878-8742-6fdb583f9173-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87ec386b-a148-4878-8742-6fdb583f9173" (UID: "87ec386b-a148-4878-8742-6fdb583f9173"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:03:47.212685 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.212664    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ec386b-a148-4878-8742-6fdb583f9173-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "87ec386b-a148-4878-8742-6fdb583f9173" (UID: "87ec386b-a148-4878-8742-6fdb583f9173"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:03:47.212774 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.212676    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ec386b-a148-4878-8742-6fdb583f9173-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "87ec386b-a148-4878-8742-6fdb583f9173" (UID: "87ec386b-a148-4878-8742-6fdb583f9173"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:03:47.214422 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.214390    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ec386b-a148-4878-8742-6fdb583f9173-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "87ec386b-a148-4878-8742-6fdb583f9173" (UID: "87ec386b-a148-4878-8742-6fdb583f9173"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:03:47.214422 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.214401    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ec386b-a148-4878-8742-6fdb583f9173-kube-api-access-xvfdf" (OuterVolumeSpecName: "kube-api-access-xvfdf") pod "87ec386b-a148-4878-8742-6fdb583f9173" (UID: "87ec386b-a148-4878-8742-6fdb583f9173"). InnerVolumeSpecName "kube-api-access-xvfdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:03:47.313880 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.313791    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87ec386b-a148-4878-8742-6fdb583f9173-cabundle-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:03:47.313880 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.313822    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87ec386b-a148-4878-8742-6fdb583f9173-proxy-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:03:47.313880 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.313832    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87ec386b-a148-4878-8742-6fdb583f9173-kserve-provision-location\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:03:47.313880 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.313841    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvfdf\" (UniqueName: \"kubernetes.io/projected/87ec386b-a148-4878-8742-6fdb583f9173-kube-api-access-xvfdf\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:03:47.313880 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.313851    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87ec386b-a148-4878-8742-6fdb583f9173-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:03:47.553392 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.553358    2573 generic.go:356] "Generic (PLEG): container finished" podID="87ec386b-a148-4878-8742-6fdb583f9173" containerID="21f035f4c3f5c9d9df99c0a8cec1a9f7c03878acd961526611386a4ec3ede64d" exitCode=0
Jul 09 18:03:47.553568 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.553419    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"
Jul 09 18:03:47.553568 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.553441    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" event={"ID":"87ec386b-a148-4878-8742-6fdb583f9173","Type":"ContainerDied","Data":"21f035f4c3f5c9d9df99c0a8cec1a9f7c03878acd961526611386a4ec3ede64d"}
Jul 09 18:03:47.553568 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.553508    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r" event={"ID":"87ec386b-a148-4878-8742-6fdb583f9173","Type":"ContainerDied","Data":"c96cdf68ab729c0f6e837e550d28d657e32af604f0bc772a4d9bb1aa35fd2cd1"}
Jul 09 18:03:47.553568 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.553547    2573 scope.go:117] "RemoveContainer" containerID="b26c00e139b5c9ec275d09d9cde654e8cd8238e93df57b0317306c69fea92c29"
Jul 09 18:03:47.562082 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.562065    2573 scope.go:117] "RemoveContainer" containerID="21f035f4c3f5c9d9df99c0a8cec1a9f7c03878acd961526611386a4ec3ede64d"
Jul 09 18:03:47.569300 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.569283    2573 scope.go:117] "RemoveContainer" containerID="006d0e30951163e483ebb145bb978e8fe7bfbba3534538afb342cca8f38b0cf3"
Jul 09 18:03:47.576297 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.576274    2573 scope.go:117] "RemoveContainer" containerID="b26c00e139b5c9ec275d09d9cde654e8cd8238e93df57b0317306c69fea92c29"
Jul 09 18:03:47.576573 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:03:47.576554    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26c00e139b5c9ec275d09d9cde654e8cd8238e93df57b0317306c69fea92c29\": container with ID starting with b26c00e139b5c9ec275d09d9cde654e8cd8238e93df57b0317306c69fea92c29 not found: ID does not exist" containerID="b26c00e139b5c9ec275d09d9cde654e8cd8238e93df57b0317306c69fea92c29"
Jul 09 18:03:47.576629 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.576582    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26c00e139b5c9ec275d09d9cde654e8cd8238e93df57b0317306c69fea92c29"} err="failed to get container status \"b26c00e139b5c9ec275d09d9cde654e8cd8238e93df57b0317306c69fea92c29\": rpc error: code = NotFound desc = could not find container \"b26c00e139b5c9ec275d09d9cde654e8cd8238e93df57b0317306c69fea92c29\": container with ID starting with b26c00e139b5c9ec275d09d9cde654e8cd8238e93df57b0317306c69fea92c29 not found: ID does not exist"
Jul 09 18:03:47.576629 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.576601    2573 scope.go:117] "RemoveContainer" containerID="21f035f4c3f5c9d9df99c0a8cec1a9f7c03878acd961526611386a4ec3ede64d"
Jul 09 18:03:47.576897 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:03:47.576874    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f035f4c3f5c9d9df99c0a8cec1a9f7c03878acd961526611386a4ec3ede64d\": container with ID starting with 21f035f4c3f5c9d9df99c0a8cec1a9f7c03878acd961526611386a4ec3ede64d not found: ID does not exist" containerID="21f035f4c3f5c9d9df99c0a8cec1a9f7c03878acd961526611386a4ec3ede64d"
Jul 09 18:03:47.576957 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.576906    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f035f4c3f5c9d9df99c0a8cec1a9f7c03878acd961526611386a4ec3ede64d"} err="failed to get container status \"21f035f4c3f5c9d9df99c0a8cec1a9f7c03878acd961526611386a4ec3ede64d\": rpc error: code = NotFound desc = could not find container \"21f035f4c3f5c9d9df99c0a8cec1a9f7c03878acd961526611386a4ec3ede64d\": container with ID starting with 21f035f4c3f5c9d9df99c0a8cec1a9f7c03878acd961526611386a4ec3ede64d not found: ID does not exist"
Jul 09 18:03:47.576957 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.576927    2573 scope.go:117] "RemoveContainer" containerID="006d0e30951163e483ebb145bb978e8fe7bfbba3534538afb342cca8f38b0cf3"
Jul 09 18:03:47.577183 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:03:47.577165    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006d0e30951163e483ebb145bb978e8fe7bfbba3534538afb342cca8f38b0cf3\": container with ID starting with 006d0e30951163e483ebb145bb978e8fe7bfbba3534538afb342cca8f38b0cf3 not found: ID does not exist" containerID="006d0e30951163e483ebb145bb978e8fe7bfbba3534538afb342cca8f38b0cf3"
Jul 09 18:03:47.577240 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.577188    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006d0e30951163e483ebb145bb978e8fe7bfbba3534538afb342cca8f38b0cf3"} err="failed to get container status \"006d0e30951163e483ebb145bb978e8fe7bfbba3534538afb342cca8f38b0cf3\": rpc error: code = NotFound desc = could not find container \"006d0e30951163e483ebb145bb978e8fe7bfbba3534538afb342cca8f38b0cf3\": container with ID starting with 006d0e30951163e483ebb145bb978e8fe7bfbba3534538afb342cca8f38b0cf3 not found: ID does not exist"
Jul 09 18:03:47.577765 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.577713    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"]
Jul 09 18:03:47.581548 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.581528    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r"]
Jul 09 18:03:47.898116 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.898034    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"]
Jul 09 18:03:47.898321 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.898308    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kserve-container"
Jul 09 18:03:47.898367 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.898322    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kserve-container"
Jul 09 18:03:47.898367 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.898332    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="storage-initializer"
Jul 09 18:03:47.898367 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.898337    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="storage-initializer"
Jul 09 18:03:47.898367 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.898349    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kube-rbac-proxy"
Jul 09 18:03:47.898367 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.898355    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kube-rbac-proxy"
Jul 09 18:03:47.898513 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.898403    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kserve-container"
Jul 09 18:03:47.898513 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.898414    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ec386b-a148-4878-8742-6fdb583f9173" containerName="kube-rbac-proxy"
Jul 09 18:03:47.902792 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.902774    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:47.909302 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.909279    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cfv7h\""
Jul 09 18:03:47.909398 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.909346    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-4d85b-kube-rbac-proxy-sar-config\""
Jul 09 18:03:47.909622 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.909600    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:03:47.909675 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.909653    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:03:47.909743 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.909706    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:03:47.910901 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.910884    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-4d85b-predictor-serving-cert\""
Jul 09 18:03:47.910963 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.910929    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:03:47.920049 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:47.920025    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"]
Jul 09 18:03:48.018617 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.018578    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ad0bc00-653f-4dac-ac58-689d70570066-cabundle-cert\") pod \"raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.018617 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.018623    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v228j\" (UniqueName: \"kubernetes.io/projected/0ad0bc00-653f-4dac-ac58-689d70570066-kube-api-access-v228j\") pod \"raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.018890 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.018647    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ad0bc00-653f-4dac-ac58-689d70570066-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.018890 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.018751    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ad0bc00-653f-4dac-ac58-689d70570066-kserve-provision-location\") pod \"raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.018890 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.018791    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ad0bc00-653f-4dac-ac58-689d70570066-proxy-tls\") pod \"raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.119547 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.119513    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ad0bc00-653f-4dac-ac58-689d70570066-proxy-tls\") pod \"raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.119762 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.119594    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ad0bc00-653f-4dac-ac58-689d70570066-cabundle-cert\") pod \"raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.119762 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.119633    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v228j\" (UniqueName: \"kubernetes.io/projected/0ad0bc00-653f-4dac-ac58-689d70570066-kube-api-access-v228j\") pod \"raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.119762 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.119659    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ad0bc00-653f-4dac-ac58-689d70570066-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.119762 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.119703    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ad0bc00-653f-4dac-ac58-689d70570066-kserve-provision-location\") pod \"raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.120151 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.120127    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ad0bc00-653f-4dac-ac58-689d70570066-kserve-provision-location\") pod \"raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.120347 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.120330    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ad0bc00-653f-4dac-ac58-689d70570066-cabundle-cert\") pod \"raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.120390 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.120351    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ad0bc00-653f-4dac-ac58-689d70570066-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.121927 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.121908    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ad0bc00-653f-4dac-ac58-689d70570066-proxy-tls\") pod \"raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.130148 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.130126    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v228j\" (UniqueName: \"kubernetes.io/projected/0ad0bc00-653f-4dac-ac58-689d70570066-kube-api-access-v228j\") pod \"raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.213198 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.213128    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:48.346198 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.346169    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"]
Jul 09 18:03:48.348405 ip-10-0-143-142 kubenswrapper[2573]: W0709 18:03:48.348378    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad0bc00_653f_4dac_ac58_689d70570066.slice/crio-27995296f5f70fc81e874b1a7b281f580ac03f7cd842eac76b3033816047871e WatchSource:0}: Error finding container 27995296f5f70fc81e874b1a7b281f580ac03f7cd842eac76b3033816047871e: Status 404 returned error can't find the container with id 27995296f5f70fc81e874b1a7b281f580ac03f7cd842eac76b3033816047871e
Jul 09 18:03:48.527533 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.527503    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87ec386b-a148-4878-8742-6fdb583f9173" path="/var/lib/kubelet/pods/87ec386b-a148-4878-8742-6fdb583f9173/volumes"
Jul 09 18:03:48.560337 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.560300    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" event={"ID":"0ad0bc00-653f-4dac-ac58-689d70570066","Type":"ContainerStarted","Data":"866ed1e5c721f2776a2d9c0b5377323bf8bfe1f27bba30d69d252cfd7b7fd711"}
Jul 09 18:03:48.560518 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:48.560363    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" event={"ID":"0ad0bc00-653f-4dac-ac58-689d70570066","Type":"ContainerStarted","Data":"27995296f5f70fc81e874b1a7b281f580ac03f7cd842eac76b3033816047871e"}
Jul 09 18:03:52.576197 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:52.576162    2573 generic.go:356] "Generic (PLEG): container finished" podID="0ad0bc00-653f-4dac-ac58-689d70570066" containerID="866ed1e5c721f2776a2d9c0b5377323bf8bfe1f27bba30d69d252cfd7b7fd711" exitCode=0
Jul 09 18:03:52.576580 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:52.576235    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" event={"ID":"0ad0bc00-653f-4dac-ac58-689d70570066","Type":"ContainerDied","Data":"866ed1e5c721f2776a2d9c0b5377323bf8bfe1f27bba30d69d252cfd7b7fd711"}
Jul 09 18:03:53.582087 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:53.582053    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" event={"ID":"0ad0bc00-653f-4dac-ac58-689d70570066","Type":"ContainerStarted","Data":"7c90ab350788be48469f3f2f87e2c1cdb7a9854e8eb33a0c96f7cca2b5197f3b"}
Jul 09 18:03:53.582445 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:53.582115    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" event={"ID":"0ad0bc00-653f-4dac-ac58-689d70570066","Type":"ContainerStarted","Data":"a46c5765b3c7a0d0df8fa16ecf6634993e8661e3b8534a722af7e59339a70177"}
Jul 09 18:03:53.582445 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:53.582306    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:53.606713 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:53.606657    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" podStartSLOduration=2.902130861 podStartE2EDuration="6.606642181s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.70451132s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:53.606301886 +0000 UTC m=+1477.619831026" watchObservedRunningTime="2026-07-09 18:03:53.606642181 +0000 UTC m=+1477.620171295"
Jul 09 18:03:54.586077 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:54.586046    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:03:54.587130 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:54.587099    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 09 18:03:55.590395 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:03:55.590359    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 09 18:04:00.594957 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:04:00.594931    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:04:00.595386 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:04:00.595363    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 09 18:04:10.596295 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:04:10.596254    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 09 18:04:16.383638 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:04:16.383599    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/1.log"
Jul 09 18:04:16.388589 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:04:16.388563    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/1.log"
Jul 09 18:04:16.390539 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:04:16.390523    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/0.log"
Jul 09 18:04:16.396493 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:04:16.396476    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/0.log"
Jul 09 18:04:20.595955 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:04:20.595916    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 09 18:04:30.596381 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:04:30.596336    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 09 18:04:40.595672 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:04:40.595587    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 09 18:04:50.596034 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:04:50.595994    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 09 18:05:00.595757 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:00.595705    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 09 18:05:10.596213 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:10.596185    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:05:17.947747 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:17.947698    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"]
Jul 09 18:05:17.948141 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:17.947955    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kserve-container" containerID="cri-o://a46c5765b3c7a0d0df8fa16ecf6634993e8661e3b8534a722af7e59339a70177" gracePeriod=30
Jul 09 18:05:17.948141 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:17.948013    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kube-rbac-proxy" containerID="cri-o://7c90ab350788be48469f3f2f87e2c1cdb7a9854e8eb33a0c96f7cca2b5197f3b" gracePeriod=30
Jul 09 18:05:18.907000 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:18.906963    2573 generic.go:356] "Generic (PLEG): container finished" podID="0ad0bc00-653f-4dac-ac58-689d70570066" containerID="7c90ab350788be48469f3f2f87e2c1cdb7a9854e8eb33a0c96f7cca2b5197f3b" exitCode=2
Jul 09 18:05:18.907183 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:18.907031    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" event={"ID":"0ad0bc00-653f-4dac-ac58-689d70570066","Type":"ContainerDied","Data":"7c90ab350788be48469f3f2f87e2c1cdb7a9854e8eb33a0c96f7cca2b5197f3b"}
Jul 09 18:05:20.591142 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:20.591093    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Jul 09 18:05:20.595364 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:20.595339    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 09 18:05:22.694331 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.694305    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:05:22.869576 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.869482    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v228j\" (UniqueName: \"kubernetes.io/projected/0ad0bc00-653f-4dac-ac58-689d70570066-kube-api-access-v228j\") pod \"0ad0bc00-653f-4dac-ac58-689d70570066\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") "
Jul 09 18:05:22.869576 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.869526    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ad0bc00-653f-4dac-ac58-689d70570066-kserve-provision-location\") pod \"0ad0bc00-653f-4dac-ac58-689d70570066\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") "
Jul 09 18:05:22.869576 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.869548    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ad0bc00-653f-4dac-ac58-689d70570066-proxy-tls\") pod \"0ad0bc00-653f-4dac-ac58-689d70570066\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") "
Jul 09 18:05:22.869843 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.869765    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ad0bc00-653f-4dac-ac58-689d70570066-kube-rbac-proxy-sar-config\") pod \"0ad0bc00-653f-4dac-ac58-689d70570066\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") "
Jul 09 18:05:22.869843 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.869817    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ad0bc00-653f-4dac-ac58-689d70570066-cabundle-cert\") pod \"0ad0bc00-653f-4dac-ac58-689d70570066\" (UID: \"0ad0bc00-653f-4dac-ac58-689d70570066\") "
Jul 09 18:05:22.869959 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.869932    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad0bc00-653f-4dac-ac58-689d70570066-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0ad0bc00-653f-4dac-ac58-689d70570066" (UID: "0ad0bc00-653f-4dac-ac58-689d70570066"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:05:22.870145 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.870115    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad0bc00-653f-4dac-ac58-689d70570066-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0ad0bc00-653f-4dac-ac58-689d70570066" (UID: "0ad0bc00-653f-4dac-ac58-689d70570066"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:05:22.870218 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.870128    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ad0bc00-653f-4dac-ac58-689d70570066-kserve-provision-location\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:05:22.870218 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.870167    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad0bc00-653f-4dac-ac58-689d70570066-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0ad0bc00-653f-4dac-ac58-689d70570066" (UID: "0ad0bc00-653f-4dac-ac58-689d70570066"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:05:22.871697 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.871675    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad0bc00-653f-4dac-ac58-689d70570066-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0ad0bc00-653f-4dac-ac58-689d70570066" (UID: "0ad0bc00-653f-4dac-ac58-689d70570066"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:05:22.871888 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.871870    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad0bc00-653f-4dac-ac58-689d70570066-kube-api-access-v228j" (OuterVolumeSpecName: "kube-api-access-v228j") pod "0ad0bc00-653f-4dac-ac58-689d70570066" (UID: "0ad0bc00-653f-4dac-ac58-689d70570066"). InnerVolumeSpecName "kube-api-access-v228j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:05:22.925196 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.925164    2573 generic.go:356] "Generic (PLEG): container finished" podID="0ad0bc00-653f-4dac-ac58-689d70570066" containerID="a46c5765b3c7a0d0df8fa16ecf6634993e8661e3b8534a722af7e59339a70177" exitCode=0
Jul 09 18:05:22.925361 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.925222    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"
Jul 09 18:05:22.925361 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.925238    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" event={"ID":"0ad0bc00-653f-4dac-ac58-689d70570066","Type":"ContainerDied","Data":"a46c5765b3c7a0d0df8fa16ecf6634993e8661e3b8534a722af7e59339a70177"}
Jul 09 18:05:22.925361 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.925287    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4" event={"ID":"0ad0bc00-653f-4dac-ac58-689d70570066","Type":"ContainerDied","Data":"27995296f5f70fc81e874b1a7b281f580ac03f7cd842eac76b3033816047871e"}
Jul 09 18:05:22.925361 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.925313    2573 scope.go:117] "RemoveContainer" containerID="7c90ab350788be48469f3f2f87e2c1cdb7a9854e8eb33a0c96f7cca2b5197f3b"
Jul 09 18:05:22.934918 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.934898    2573 scope.go:117] "RemoveContainer" containerID="a46c5765b3c7a0d0df8fa16ecf6634993e8661e3b8534a722af7e59339a70177"
Jul 09 18:05:22.942343 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.942325    2573 scope.go:117] "RemoveContainer" containerID="866ed1e5c721f2776a2d9c0b5377323bf8bfe1f27bba30d69d252cfd7b7fd711"
Jul 09 18:05:22.950000 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.949971    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"]
Jul 09 18:05:22.950781 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.950713    2573 scope.go:117] "RemoveContainer" containerID="7c90ab350788be48469f3f2f87e2c1cdb7a9854e8eb33a0c96f7cca2b5197f3b"
Jul 09 18:05:22.951088 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:05:22.951062    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c90ab350788be48469f3f2f87e2c1cdb7a9854e8eb33a0c96f7cca2b5197f3b\": container with ID starting with 7c90ab350788be48469f3f2f87e2c1cdb7a9854e8eb33a0c96f7cca2b5197f3b not found: ID does not exist" containerID="7c90ab350788be48469f3f2f87e2c1cdb7a9854e8eb33a0c96f7cca2b5197f3b"
Jul 09 18:05:22.951173 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.951092    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c90ab350788be48469f3f2f87e2c1cdb7a9854e8eb33a0c96f7cca2b5197f3b"} err="failed to get container status \"7c90ab350788be48469f3f2f87e2c1cdb7a9854e8eb33a0c96f7cca2b5197f3b\": rpc error: code = NotFound desc = could not find container \"7c90ab350788be48469f3f2f87e2c1cdb7a9854e8eb33a0c96f7cca2b5197f3b\": container with ID starting with 7c90ab350788be48469f3f2f87e2c1cdb7a9854e8eb33a0c96f7cca2b5197f3b not found: ID does not exist"
Jul 09 18:05:22.951173 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.951113    2573 scope.go:117] "RemoveContainer" containerID="a46c5765b3c7a0d0df8fa16ecf6634993e8661e3b8534a722af7e59339a70177"
Jul 09 18:05:22.951383 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:05:22.951363    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a46c5765b3c7a0d0df8fa16ecf6634993e8661e3b8534a722af7e59339a70177\": container with ID starting with a46c5765b3c7a0d0df8fa16ecf6634993e8661e3b8534a722af7e59339a70177 not found: ID does not exist" containerID="a46c5765b3c7a0d0df8fa16ecf6634993e8661e3b8534a722af7e59339a70177"
Jul 09 18:05:22.951444 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.951390    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46c5765b3c7a0d0df8fa16ecf6634993e8661e3b8534a722af7e59339a70177"} err="failed to get container status \"a46c5765b3c7a0d0df8fa16ecf6634993e8661e3b8534a722af7e59339a70177\": rpc error: code = NotFound desc = could not find container \"a46c5765b3c7a0d0df8fa16ecf6634993e8661e3b8534a722af7e59339a70177\": container with ID starting with a46c5765b3c7a0d0df8fa16ecf6634993e8661e3b8534a722af7e59339a70177 not found: ID does not exist"
Jul 09 18:05:22.951444 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.951405    2573 scope.go:117] "RemoveContainer" containerID="866ed1e5c721f2776a2d9c0b5377323bf8bfe1f27bba30d69d252cfd7b7fd711"
Jul 09 18:05:22.951661 ip-10-0-143-142 kubenswrapper[2573]: E0709 18:05:22.951642    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866ed1e5c721f2776a2d9c0b5377323bf8bfe1f27bba30d69d252cfd7b7fd711\": container with ID starting with 866ed1e5c721f2776a2d9c0b5377323bf8bfe1f27bba30d69d252cfd7b7fd711 not found: ID does not exist" containerID="866ed1e5c721f2776a2d9c0b5377323bf8bfe1f27bba30d69d252cfd7b7fd711"
Jul 09 18:05:22.951768 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.951664    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866ed1e5c721f2776a2d9c0b5377323bf8bfe1f27bba30d69d252cfd7b7fd711"} err="failed to get container status \"866ed1e5c721f2776a2d9c0b5377323bf8bfe1f27bba30d69d252cfd7b7fd711\": rpc error: code = NotFound desc = could not find container \"866ed1e5c721f2776a2d9c0b5377323bf8bfe1f27bba30d69d252cfd7b7fd711\": container with ID starting with 866ed1e5c721f2776a2d9c0b5377323bf8bfe1f27bba30d69d252cfd7b7fd711 not found: ID does not exist"
Jul 09 18:05:22.955942 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.955917    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4"]
Jul 09 18:05:22.971247 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.971221    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v228j\" (UniqueName: \"kubernetes.io/projected/0ad0bc00-653f-4dac-ac58-689d70570066-kube-api-access-v228j\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:05:22.971247 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.971245    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ad0bc00-653f-4dac-ac58-689d70570066-proxy-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:05:22.971426 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.971257    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ad0bc00-653f-4dac-ac58-689d70570066-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:05:22.971426 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:22.971265    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ad0bc00-653f-4dac-ac58-689d70570066-cabundle-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 09 18:05:24.527109 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:24.527072    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" path="/var/lib/kubelet/pods/0ad0bc00-653f-4dac-ac58-689d70570066/volumes"
Jul 09 18:05:54.704004 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.703970    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rgx2c/must-gather-fcpt8"]
Jul 09 18:05:54.704473 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.704237    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kserve-container"
Jul 09 18:05:54.704473 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.704248    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kserve-container"
Jul 09 18:05:54.704473 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.704263    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kube-rbac-proxy"
Jul 09 18:05:54.704473 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.704268    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kube-rbac-proxy"
Jul 09 18:05:54.704473 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.704279    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="storage-initializer"
Jul 09 18:05:54.704473 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.704284    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="storage-initializer"
Jul 09 18:05:54.704473 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.704334    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kserve-container"
Jul 09 18:05:54.704473 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.704343    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ad0bc00-653f-4dac-ac58-689d70570066" containerName="kube-rbac-proxy"
Jul 09 18:05:54.706842 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.706825    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rgx2c/must-gather-fcpt8"
Jul 09 18:05:54.709961 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.709935    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rgx2c\"/\"kube-root-ca.crt\""
Jul 09 18:05:54.709961 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.709958    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rgx2c\"/\"openshift-service-ca.crt\""
Jul 09 18:05:54.711408 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.711393    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rgx2c\"/\"default-dockercfg-srsdn\""
Jul 09 18:05:54.715027 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.715007    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rgx2c/must-gather-fcpt8"]
Jul 09 18:05:54.812321 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.812281    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6269\" (UniqueName: \"kubernetes.io/projected/c3227b25-aa12-4058-bacd-4947d279da83-kube-api-access-w6269\") pod \"must-gather-fcpt8\" (UID: \"c3227b25-aa12-4058-bacd-4947d279da83\") " pod="openshift-must-gather-rgx2c/must-gather-fcpt8"
Jul 09 18:05:54.812494 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.812389    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c3227b25-aa12-4058-bacd-4947d279da83-must-gather-output\") pod \"must-gather-fcpt8\" (UID: \"c3227b25-aa12-4058-bacd-4947d279da83\") " pod="openshift-must-gather-rgx2c/must-gather-fcpt8"
Jul 09 18:05:54.913570 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.913536    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c3227b25-aa12-4058-bacd-4947d279da83-must-gather-output\") pod \"must-gather-fcpt8\" (UID: \"c3227b25-aa12-4058-bacd-4947d279da83\") " pod="openshift-must-gather-rgx2c/must-gather-fcpt8"
Jul 09 18:05:54.913754 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.913582    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6269\" (UniqueName: \"kubernetes.io/projected/c3227b25-aa12-4058-bacd-4947d279da83-kube-api-access-w6269\") pod \"must-gather-fcpt8\" (UID: \"c3227b25-aa12-4058-bacd-4947d279da83\") " pod="openshift-must-gather-rgx2c/must-gather-fcpt8"
Jul 09 18:05:54.913954 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.913932    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c3227b25-aa12-4058-bacd-4947d279da83-must-gather-output\") pod \"must-gather-fcpt8\" (UID: \"c3227b25-aa12-4058-bacd-4947d279da83\") " pod="openshift-must-gather-rgx2c/must-gather-fcpt8"
Jul 09 18:05:54.922222 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:54.922198    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6269\" (UniqueName: \"kubernetes.io/projected/c3227b25-aa12-4058-bacd-4947d279da83-kube-api-access-w6269\") pod \"must-gather-fcpt8\" (UID: \"c3227b25-aa12-4058-bacd-4947d279da83\") " pod="openshift-must-gather-rgx2c/must-gather-fcpt8"
Jul 09 18:05:55.016960 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:55.016932    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rgx2c/must-gather-fcpt8"
Jul 09 18:05:55.146611 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:55.146548    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rgx2c/must-gather-fcpt8"]
Jul 09 18:05:55.148709 ip-10-0-143-142 kubenswrapper[2573]: W0709 18:05:55.148683    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3227b25_aa12_4058_bacd_4947d279da83.slice/crio-47471b33a457fec9c83f42d9b30a026988fe09fc9284cbbbfd177fa16718f3e0 WatchSource:0}: Error finding container 47471b33a457fec9c83f42d9b30a026988fe09fc9284cbbbfd177fa16718f3e0: Status 404 returned error can't find the container with id 47471b33a457fec9c83f42d9b30a026988fe09fc9284cbbbfd177fa16718f3e0
Jul 09 18:05:55.150634 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:55.150620    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:05:56.054768 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:56.054713    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgx2c/must-gather-fcpt8" event={"ID":"c3227b25-aa12-4058-bacd-4947d279da83","Type":"ContainerStarted","Data":"6876da9ea641d1ed98fee9d89ff9c61bb3ce5bf5b582fb05395ca115b14cd1a9"}
Jul 09 18:05:56.055184 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:56.054800    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgx2c/must-gather-fcpt8" event={"ID":"c3227b25-aa12-4058-bacd-4947d279da83","Type":"ContainerStarted","Data":"47471b33a457fec9c83f42d9b30a026988fe09fc9284cbbbfd177fa16718f3e0"}
Jul 09 18:05:57.061931 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:57.061886    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgx2c/must-gather-fcpt8" event={"ID":"c3227b25-aa12-4058-bacd-4947d279da83","Type":"ContainerStarted","Data":"2e7d42182820c80271268c8b381bb63f8119a25e57d62634b21e5cb9a6d48cc5"}
Jul 09 18:05:57.080590 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:57.080532    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rgx2c/must-gather-fcpt8" podStartSLOduration=2.291475569 podStartE2EDuration="3.080515465s" totalImagesPullingTime="789.039896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:57.079336792 +0000 UTC m=+1601.092865907" watchObservedRunningTime="2026-07-09 18:05:57.080515465 +0000 UTC m=+1601.094044578"
Jul 09 18:05:57.482923 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:57.482839    2573 ???:1] "http: TLS handshake error from 10.0.128.245:36888: EOF"
Jul 09 18:05:57.487738 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:57.487655    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mcsfz_b282b3a8-fc56-4bfc-8626-947aada8ecde/global-pull-secret-syncer/0.log"
Jul 09 18:05:57.599960 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:57.599915    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ps4sv_26c0b0ef-78e7-455e-b5cf-52376c022784/konnectivity-agent/0.log"
Jul 09 18:05:57.665360 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:05:57.665324    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-142.ec2.internal_0b3b8930aa1846297e408b672c61a6ab/haproxy/0.log"
Jul 09 18:06:01.926024 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:01.925986    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-wp6zp_de82e36c-3a74-425b-b950-6983d4686d0a/cluster-monitoring-operator/0.log"
Jul 09 18:06:01.955697 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:01.955667    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-dnq47_87a6d039-68e6-4655-bf1f-9552c7d5b789/kube-state-metrics/0.log"
Jul 09 18:06:01.976043 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:01.975990    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-dnq47_87a6d039-68e6-4655-bf1f-9552c7d5b789/kube-rbac-proxy-main/0.log"
Jul 09 18:06:02.002500 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:02.002408    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-dnq47_87a6d039-68e6-4655-bf1f-9552c7d5b789/kube-rbac-proxy-self/0.log"
Jul 09 18:06:02.087477 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:02.087449    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ms8ps_2f8c82b7-be76-45ff-8440-eddb547acf3b/node-exporter/0.log"
Jul 09 18:06:02.109935 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:02.109905    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ms8ps_2f8c82b7-be76-45ff-8440-eddb547acf3b/kube-rbac-proxy/0.log"
Jul 09 18:06:02.138399 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:02.138372    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ms8ps_2f8c82b7-be76-45ff-8440-eddb547acf3b/init-textfile/0.log"
Jul 09 18:06:04.725054 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:04.725002    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/1.log"
Jul 09 18:06:04.731420 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:04.731385    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mj9vb_9604ff70-0e20-445a-94a5-ff74c0c9579c/console-operator/2.log"
Jul 09 18:06:05.638171 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.638139    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"]
Jul 09 18:06:05.645871 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.645839    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.656349 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.656321    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"]
Jul 09 18:06:05.721712 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.721676    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6c8470c9-d4b9-4c76-9755-9f2822d6cc88-podres\") pod \"perf-node-gather-daemonset-lvrsw\" (UID: \"6c8470c9-d4b9-4c76-9755-9f2822d6cc88\") " pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.722039 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.722014    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6c8470c9-d4b9-4c76-9755-9f2822d6cc88-proc\") pod \"perf-node-gather-daemonset-lvrsw\" (UID: \"6c8470c9-d4b9-4c76-9755-9f2822d6cc88\") " pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.722184 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.722162    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c8470c9-d4b9-4c76-9755-9f2822d6cc88-lib-modules\") pod \"perf-node-gather-daemonset-lvrsw\" (UID: \"6c8470c9-d4b9-4c76-9755-9f2822d6cc88\") " pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.722315 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.722207    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsjvp\" (UniqueName: \"kubernetes.io/projected/6c8470c9-d4b9-4c76-9755-9f2822d6cc88-kube-api-access-tsjvp\") pod \"perf-node-gather-daemonset-lvrsw\" (UID: \"6c8470c9-d4b9-4c76-9755-9f2822d6cc88\") " pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.722315 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.722257    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c8470c9-d4b9-4c76-9755-9f2822d6cc88-sys\") pod \"perf-node-gather-daemonset-lvrsw\" (UID: \"6c8470c9-d4b9-4c76-9755-9f2822d6cc88\") " pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.761699 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.761667    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-lqj4g_4faab181-b8e3-4a28-bc8f-0c59f5e55edd/volume-data-source-validator/0.log"
Jul 09 18:06:05.822994 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.822953    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsjvp\" (UniqueName: \"kubernetes.io/projected/6c8470c9-d4b9-4c76-9755-9f2822d6cc88-kube-api-access-tsjvp\") pod \"perf-node-gather-daemonset-lvrsw\" (UID: \"6c8470c9-d4b9-4c76-9755-9f2822d6cc88\") " pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.823274 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.823256    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c8470c9-d4b9-4c76-9755-9f2822d6cc88-sys\") pod \"perf-node-gather-daemonset-lvrsw\" (UID: \"6c8470c9-d4b9-4c76-9755-9f2822d6cc88\") " pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.823462 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.823337    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c8470c9-d4b9-4c76-9755-9f2822d6cc88-sys\") pod \"perf-node-gather-daemonset-lvrsw\" (UID: \"6c8470c9-d4b9-4c76-9755-9f2822d6cc88\") " pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.823462 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.823445    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6c8470c9-d4b9-4c76-9755-9f2822d6cc88-podres\") pod \"perf-node-gather-daemonset-lvrsw\" (UID: \"6c8470c9-d4b9-4c76-9755-9f2822d6cc88\") " pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.823767 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.823744    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6c8470c9-d4b9-4c76-9755-9f2822d6cc88-podres\") pod \"perf-node-gather-daemonset-lvrsw\" (UID: \"6c8470c9-d4b9-4c76-9755-9f2822d6cc88\") " pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.824002 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.823910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6c8470c9-d4b9-4c76-9755-9f2822d6cc88-proc\") pod \"perf-node-gather-daemonset-lvrsw\" (UID: \"6c8470c9-d4b9-4c76-9755-9f2822d6cc88\") " pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.824151 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.824132    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c8470c9-d4b9-4c76-9755-9f2822d6cc88-lib-modules\") pod \"perf-node-gather-daemonset-lvrsw\" (UID: \"6c8470c9-d4b9-4c76-9755-9f2822d6cc88\") " pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.824431 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.823961    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6c8470c9-d4b9-4c76-9755-9f2822d6cc88-proc\") pod \"perf-node-gather-daemonset-lvrsw\" (UID: \"6c8470c9-d4b9-4c76-9755-9f2822d6cc88\") " pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.824515 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.824340    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c8470c9-d4b9-4c76-9755-9f2822d6cc88-lib-modules\") pod \"perf-node-gather-daemonset-lvrsw\" (UID: \"6c8470c9-d4b9-4c76-9755-9f2822d6cc88\") " pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.832540 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.832521    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsjvp\" (UniqueName: \"kubernetes.io/projected/6c8470c9-d4b9-4c76-9755-9f2822d6cc88-kube-api-access-tsjvp\") pod \"perf-node-gather-daemonset-lvrsw\" (UID: \"6c8470c9-d4b9-4c76-9755-9f2822d6cc88\") " pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:05.960751 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:05.960641    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:06.137750 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:06.137659    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"]
Jul 09 18:06:06.143531 ip-10-0-143-142 kubenswrapper[2573]: W0709 18:06:06.143498    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6c8470c9_d4b9_4c76_9755_9f2822d6cc88.slice/crio-488bda2d6177453399ff1489fc4818479d947f72ed2a29947d74fc2df8b72401 WatchSource:0}: Error finding container 488bda2d6177453399ff1489fc4818479d947f72ed2a29947d74fc2df8b72401: Status 404 returned error can't find the container with id 488bda2d6177453399ff1489fc4818479d947f72ed2a29947d74fc2df8b72401
Jul 09 18:06:06.722533 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:06.722505    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-675dp_c71f2394-9aa0-40bf-a07c-26cd48a93a1c/dns/0.log"
Jul 09 18:06:06.748922 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:06.748899    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-675dp_c71f2394-9aa0-40bf-a07c-26cd48a93a1c/kube-rbac-proxy/0.log"
Jul 09 18:06:06.879139 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:06.879106    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cctw4_a3dcc235-2a55-4e67-a42c-4d047739f8f0/dns-node-resolver/0.log"
Jul 09 18:06:07.137615 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:07.137569    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw" event={"ID":"6c8470c9-d4b9-4c76-9755-9f2822d6cc88","Type":"ContainerStarted","Data":"9a43fae22ca0ab11333b6a611b3968382262956411c89ebaa390b044dba1efa8"}
Jul 09 18:06:07.137816 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:07.137643    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw" event={"ID":"6c8470c9-d4b9-4c76-9755-9f2822d6cc88","Type":"ContainerStarted","Data":"488bda2d6177453399ff1489fc4818479d947f72ed2a29947d74fc2df8b72401"}
Jul 09 18:06:07.137962 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:07.137941    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:07.158291 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:07.158242    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw" podStartSLOduration=2.158228551 podStartE2EDuration="2.158228551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:07.156492791 +0000 UTC m=+1611.170021927" watchObservedRunningTime="2026-07-09 18:06:07.158228551 +0000 UTC m=+1611.171757665"
Jul 09 18:06:07.444934 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:07.444861    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jbqs8_76f015b9-8aaa-40ee-a73e-492fb91418d0/node-ca/0.log"
Jul 09 18:06:08.302946 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:08.302876    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d6b67c788-g98z2_d6450393-0509-4bb5-b799-9c97d8ce5c9e/router/0.log"
Jul 09 18:06:08.675986 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:08.675909    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9rw5k_00a8e0bf-d0bf-4c7b-9f33-d7dde97ba517/serve-healthcheck-canary/0.log"
Jul 09 18:06:09.067926 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:09.067901    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-xkn6j_1a2bee9d-5d0f-4c62-a69b-b8e7993d858a/insights-operator/1.log"
Jul 09 18:06:09.068247 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:09.068233    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-xkn6j_1a2bee9d-5d0f-4c62-a69b-b8e7993d858a/insights-operator/0.log"
Jul 09 18:06:09.087878 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:09.087854    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-54d9n_81dade0c-92b2-44a3-9b6d-5ba9265259a3/kube-rbac-proxy/0.log"
Jul 09 18:06:09.109759 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:09.109732    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-54d9n_81dade0c-92b2-44a3-9b6d-5ba9265259a3/exporter/0.log"
Jul 09 18:06:09.134307 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:09.134287    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-54d9n_81dade0c-92b2-44a3-9b6d-5ba9265259a3/extractor/0.log"
Jul 09 18:06:11.365354 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:11.365325    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-jxl42_e63e97aa-3db3-4aae-825b-926ea7955787/server/0.log"
Jul 09 18:06:11.527228 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:11.527199    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-88p49_01a32118-e030-4160-ae6e-c44b733e6370/s3-init/0.log"
Jul 09 18:06:11.543054 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:11.543034    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-88p49_01a32118-e030-4160-ae6e-c44b733e6370/download-hf-model/0.log"
Jul 09 18:06:13.151253 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:13.151228    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rgx2c/perf-node-gather-daemonset-lvrsw"
Jul 09 18:06:16.006524 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:16.006491    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-nmqt5_5fb00f50-6d74-4c8c-9a7e-310f46e12b0f/kube-storage-version-migrator-operator/1.log"
Jul 09 18:06:16.007476 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:16.007456    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-nmqt5_5fb00f50-6d74-4c8c-9a7e-310f46e12b0f/kube-storage-version-migrator-operator/0.log"
Jul 09 18:06:17.342077 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:17.342053    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-glm8m_6fb10245-3407-451c-a7fc-b4a6948ffb01/kube-multus-additional-cni-plugins/0.log"
Jul 09 18:06:17.361579 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:17.361560    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-glm8m_6fb10245-3407-451c-a7fc-b4a6948ffb01/egress-router-binary-copy/0.log"
Jul 09 18:06:17.381229 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:17.381206    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-glm8m_6fb10245-3407-451c-a7fc-b4a6948ffb01/cni-plugins/0.log"
Jul 09 18:06:17.400715 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:17.400647    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-glm8m_6fb10245-3407-451c-a7fc-b4a6948ffb01/bond-cni-plugin/0.log"
Jul 09 18:06:17.419515 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:17.419494    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-glm8m_6fb10245-3407-451c-a7fc-b4a6948ffb01/routeoverride-cni/0.log"
Jul 09 18:06:17.439709 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:17.439691    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-glm8m_6fb10245-3407-451c-a7fc-b4a6948ffb01/whereabouts-cni-bincopy/0.log"
Jul 09 18:06:17.459211 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:17.459185    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-glm8m_6fb10245-3407-451c-a7fc-b4a6948ffb01/whereabouts-cni/0.log"
Jul 09 18:06:17.488548 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:17.488531    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ds27w_46f1c9d1-6578-4343-88ba-900a5cc5bd28/kube-multus/0.log"
Jul 09 18:06:17.576349 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:17.576324    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6qvsq_8a62897b-ea47-4b2f-8762-a4fc7b7a18dc/network-metrics-daemon/0.log"
Jul 09 18:06:17.633453 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:17.633422    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6qvsq_8a62897b-ea47-4b2f-8762-a4fc7b7a18dc/kube-rbac-proxy/0.log"
Jul 09 18:06:18.471139 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:18.471109    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-controller/0.log"
Jul 09 18:06:18.489149 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:18.489124    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/1.log"
Jul 09 18:06:18.489350 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:18.489238    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovn-acl-logging/0.log"
Jul 09 18:06:18.517030 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:18.517004    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/kube-rbac-proxy-node/0.log"
Jul 09 18:06:18.541772 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:18.541734    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/kube-rbac-proxy-ovn-metrics/0.log"
Jul 09 18:06:18.562073 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:18.562051    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/northd/0.log"
Jul 09 18:06:18.586030 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:18.586000    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/nbdb/0.log"
Jul 09 18:06:18.610259 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:18.610243    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/sbdb/0.log"
Jul 09 18:06:18.656971 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:18.656934    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-64kgs_fd0f26b5-5eac-453c-ba70-2cdef4bd156d/ovnkube-controller/0.log"
Jul 09 18:06:20.122348 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:20.122311    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-869dcfcdcd-dkxlh_04d0d87c-7e14-4ad2-8e16-b125198c2820/check-endpoints/0.log"
Jul 09 18:06:20.193680 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:20.193652    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-wtdtc_fb0f43af-cdd0-41f4-bedb-ba61bdfd16ae/network-check-target-container/0.log"
Jul 09 18:06:21.129190 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:21.129157    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-7gvh8_fcc10c4d-ba3f-4dd7-90ca-02826f1366b9/iptables-alerter/0.log"
Jul 09 18:06:21.967694 ip-10-0-143-142 kubenswrapper[2573]: I0709 18:06:21.967667    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-rnsm2_feb7242f-ff85-407e-a02d-edca7d9241d1/tuned/0.log"