Jul 07 14:57:18.135645 ip-10-0-143-168 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 14:57:18.540158 ip-10-0-143-168 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 14:57:18.540158 ip-10-0-143-168 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 14:57:18.540158 ip-10-0-143-168 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 14:57:18.540158 ip-10-0-143-168 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 14:57:18.540158 ip-10-0-143-168 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 14:57:18.540158 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.534876    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 14:57:18.540158 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.539973    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:57:18.540158 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.539985    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:57:18.540158 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.539989    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:57:18.540158 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.539992    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:57:18.540158 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.539995    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:57:18.540158 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.539998    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:57:18.540158 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540001    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:57:18.540158 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540004    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540007    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540009    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540012    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540015    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540017    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540021    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540023    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540026    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540028    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540031    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540033    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540036    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540038    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540041    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540044    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540046    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540049    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540051    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:57:18.541007 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540054    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540057    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540060    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540063    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540066    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540068    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540071    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540073    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540076    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540079    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540081    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540084    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540087    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540089    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540093    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540096    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540098    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540101    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540103    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540106    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:57:18.541644 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540109    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540111    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540114    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540117    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540120    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540122    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540125    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540127    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540130    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540133    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540135    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540138    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540140    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540145    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540149    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540152    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540156    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540158    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540161    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:57:18.542283 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540164    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540166    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540169    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540172    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540174    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540177    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540180    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540185    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540189    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540192    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540195    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540198    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540201    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540204    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540207    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540210    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540212    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540215    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540217    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:57:18.542906 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540221    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540225    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540228    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540230    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540233    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540236    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540239    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540243    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540246    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540248    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540251    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540254    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540257    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540260    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540262    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.540993    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541000    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541004    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541008    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541012    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:57:18.543528 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541017    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541021    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541025    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541029    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541032    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541036    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541039    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541042    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541044    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541047    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541049    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541052    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541055    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541058    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541060    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541063    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541065    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541068    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541070    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541073    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:57:18.544154 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541075    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541078    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541080    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541083    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541086    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541089    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541091    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541094    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541097    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541099    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541101    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541104    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541107    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541110    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541112    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541115    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541117    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541120    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541123    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541126    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:57:18.544800 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541128    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541131    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541133    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541136    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541139    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541142    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541144    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541147    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541149    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541152    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541154    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541157    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541159    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541162    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541164    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541167    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541169    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541172    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541174    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541176    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:57:18.545633 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541180    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541182    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541185    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541187    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541190    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541193    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541196    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541199    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541202    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541204    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541207    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541209    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541212    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541215    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541217    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541220    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541222    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541225    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541227    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:57:18.546239 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541230    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541232    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541235    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541237    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541240    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541243    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541245    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541248    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541253    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541255    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541258    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541261    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541263    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541267    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541270    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541672    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541678    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541682    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541685    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541689    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:57:18.546854 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541692    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541695    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541698    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541700    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541703    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541705    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541708    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541711    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541713    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541715    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541718    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541720    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541723    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541726    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541730    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541733    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541737    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541739    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541742    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:57:18.547470 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541745    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541751    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541754    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541757    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541760    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541762    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541765    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541767    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541770    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541772    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541775    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541778    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541780    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541783    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541786    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541788    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541791    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541794    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541797    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:57:18.548100 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541799    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541802    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541806    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541808    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541811    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541814    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541816    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541819    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541821    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541824    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541826    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541829    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541831    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541833    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541837    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541840    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541843    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541845    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541848    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541850    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:57:18.548594 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541853    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541856    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541858    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541861    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541863    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541866    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541868    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541872    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541874    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541877    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541879    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541884    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541887    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541890    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541893    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541896    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541898    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541902    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541905    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:57:18.549151 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541908    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541910    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541913    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541916    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541918    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541920    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541923    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541927    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541929    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541932    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541935    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541937    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541940    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541942    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541945    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541947    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.541950    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542018    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542025    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542031    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542035    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 14:57:18.549756 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542040    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542043    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542048    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542052    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542055    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542058    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542061    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542065    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542068    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542073    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542076    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542079    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542082    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542085    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542089    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542092    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542095    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542098    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542103    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542107    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542110    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542113    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542116    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542119    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 14:57:18.550611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542122    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542125    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542128    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542131    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542135    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542138    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542141    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542143    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542146    2581 flags.go:64] FLAG: --enable-server="true"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542150    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542154    2581 flags.go:64] FLAG: --event-burst="100"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542157    2581 flags.go:64] FLAG: --event-qps="50"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542160    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542163    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542165    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542169    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542171    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542175    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542178    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542181    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542184    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542187    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542190    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542193    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542196    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 14:57:18.551254 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542199    2581 flags.go:64] FLAG: --feature-gates=""
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542204    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542207    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542210    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542213    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542215    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542219    2581 flags.go:64] FLAG: --help="false"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542221    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-143-168.ec2.internal"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542224    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542227    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542230    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542234    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542237    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542240    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542242    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542245    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542248    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542252    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542255    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542257    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542260    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542263    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542266    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542269    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 14:57:18.552100 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542273    2581 flags.go:64] FLAG: --lock-file=""
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542276    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542279    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542282    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542287    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542290    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542293    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542296    2581 flags.go:64] FLAG: --logging-format="text"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542299    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542303    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542306    2581 flags.go:64] FLAG: --manifest-url=""
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542308    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542313    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542316    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542319    2581 flags.go:64] FLAG: --max-pods="110"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542322    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542325    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542328    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542331    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542334    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542337    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542339    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542359    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542362    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542365    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 14:57:18.555583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542368    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542371    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542377    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542380    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542383    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542385    2581 flags.go:64] FLAG: --port="10250"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542388    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542392    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ac392ebd1231d191"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542396    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542399    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542401    2581 flags.go:64] FLAG: --register-node="true"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542404    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542408    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542411    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542414    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542417    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542420    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542423    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542427    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542430    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542433    2581 flags.go:64] FLAG: --runonce="false"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542436    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542439    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542442    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542445    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 14:57:18.556275 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542447    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542450    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542453    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542456    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542459    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542462    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542465    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542468    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542470    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542474    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542477    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542482    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542485    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542487    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542493    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542496    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542499    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542502    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542505    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542508    2581 flags.go:64] FLAG: --v="2"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542511    2581 flags.go:64] FLAG: --version="false"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542515    2581 flags.go:64] FLAG: --vmodule=""
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542519    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.542522    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542633    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:57:18.661588 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542637    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:57:18.645151 ip-10-0-143-168 systemd[1]: Started Kubernetes Kubelet.
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542640    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542643    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542646    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542649    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542652    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542655    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542657    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542660    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542662    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542665    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542668    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542670    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542673    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542676    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542678    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542680    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542684    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542688    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:57:18.662540 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542691    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542696    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542700    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542704    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542707    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542709    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542712    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542715    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542718    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542721    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542724    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542726    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542729    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542732    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542735    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542737    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542740    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542743    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542746    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542749    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:57:18.663339 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542752    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542754    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542757    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542759    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542762    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542764    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542767    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542769    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542772    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542774    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542776    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542779    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542782    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542786    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542790    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542793    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542795    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542798    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542800    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:57:18.664129 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542803    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542806    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542808    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542812    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542815    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542818    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542820    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542823    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542825    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542828    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542831    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542833    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542836    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542839    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542841    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542844    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542846    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542849    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542851    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:57:18.664786 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542853    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542856    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542858    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542861    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542863    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542866    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542869    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542872    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542876    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542878    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542881    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542883    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542886    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542888    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542891    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542893    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542896    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542898    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542902    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542906    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:57:18.665417 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.542908    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:57:18.665967 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.543627    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 14:57:18.665967 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.561464    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 14:57:18.665967 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.561489    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 14:57:18.665967 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.561510    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 14:57:18.665967 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.561520    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 14:57:18.665967 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561579    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:57:18.665967 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561584    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:57:18.665967 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561588    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:57:18.665967 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561592    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:57:18.665967 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561595    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:57:18.665967 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561598    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:57:18.665967 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561601    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:57:18.665967 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561604    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:57:18.665967 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561606    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:57:18.665967 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561609    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561614    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561618    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561621    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561624    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561628    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561632    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561636    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561640    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561643    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561646    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561649    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561652    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561656    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561658    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561662    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561665    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561668    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561670    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:57:18.666444 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561673    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561676    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561680    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561685    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561687    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561690    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561693    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561695    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561698    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561701    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561704    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561706    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561709    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561712    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561715    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561717    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561720    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561723    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561726    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561729    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:57:18.666955 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561731    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561734    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561736    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561739    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561741    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561744    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561746    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561749    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561752    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561754    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561757    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561759    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561762    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561765    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561768    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561771    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561773    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561776    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561779    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561781    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:57:18.667542 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561783    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561786    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561789    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561792    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561795    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561797    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561800    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561802    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561805    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561808    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561810    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561813    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561816    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561819    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561821    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561824    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561826    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561829    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561831    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:57:18.668074 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561834    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:57:18.668626 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561836    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:57:18.668626 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561839    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:57:18.668626 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561842    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:57:18.668626 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561845    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:57:18.668626 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561847    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:57:18.668626 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561850    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:57:18.668626 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561853    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:57:18.668626 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561856    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:57:18.668626 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561858    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:57:18.668626 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561861    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:57:18.668626 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561864    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:57:18.668626 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.561869    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 14:57:18.668626 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561987    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:57:18.668626 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561993    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:57:18.668626 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561996    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.561999    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562002    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562004    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562007    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562010    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562012    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562015    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562017    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562020    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562023    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562026    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562029    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562032    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562034    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562037    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562040    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562042    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562045    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562047    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:57:18.669038 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562050    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562052    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562055    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562057    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562060    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562063    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562065    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562068    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562070    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562073    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562076    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562081    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562084    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562086    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562090    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562092    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562095    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562098    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562100    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562103    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:57:18.670036 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562105    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562109    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562112    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562115    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562118    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562121    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562123    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562125    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562128    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562131    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562133    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562136    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562139    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562142    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562146    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562149    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562152    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562155    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562158    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:57:18.671009 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562160    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562163    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562166    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562171    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562173    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562176    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562178    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562181    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562184    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562186    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562188    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562191    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562194    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562196    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562199    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562203    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562206    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562210    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562213    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:57:18.672017 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562216    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562219    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562221    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562224    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562226    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562229    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562231    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562234    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562236    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562240    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562243    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562246    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562248    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562251    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562253    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562256    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562260    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562263    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:57:18.672909 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:18.562266    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:57:18.673569 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.562271    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 14:57:18.673569 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.562416    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 14:57:18.673569 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.565442    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 14:57:18.673569 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.574101    2581 server.go:1019] "Starting client certificate rotation"
Jul 07 14:57:18.673569 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.574209    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 14:57:18.673569 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.574239    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 14:57:18.673569 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.577259    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 14:57:18.673569 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.578613    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 14:57:18.673569 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.599896    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 07 14:57:18.673569 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.602108    2581 log.go:25] "Validated CRI v1 image API"
Jul 07 14:57:18.673569 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.603593    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 14:57:18.673569 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.606056    2581 fs.go:135] Filesystem UUIDs: map[38fda68b-9e36-4854-ad5e-8633bf95d081:/dev/nvme0n1p3 4eeb7f15-3bc8-4b18-87ed-bca7e94c38c3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 07 14:57:18.673986 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.606072    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 14:57:18.673986 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.607655    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.617694    2581 manager.go:216] Machine: {Timestamp:2026-07-07 14:57:18.610758585 +0000 UTC m=+0.343370897 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100689 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27e5e3ac98b752bf0288a2d331a82a SystemUUID:ec27e5e3-ac98-b752-bf02-88a2d331a82a BootID:a4c07a5e-b850-427f-80f0-89479bb90677 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:07:96:07:37:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:07:96:07:37:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:cf:ce:f5:88:c0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.617806    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.617893    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.620885    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.620914    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-168.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.621717    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.621727    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.621734    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.622409    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.624008    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.633431    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.633555    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.633595    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.633607    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.633625    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.633635    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.633643    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 14:57:18.676847 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.634016    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.634032    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.636223    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tk7tq"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.636845    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.639331    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.639393    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.640555    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.640569    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.640576    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.640581    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.640591    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.640597    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.640606    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.640612    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.640618    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.640624    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.640632    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.640637    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.641397    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.641407    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.644560    2581 server.go:1268] "Started kubelet"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.644885    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.652235    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tk7tq"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.652273    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-168.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:18.652386    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-168.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.652441    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:18.652501    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.652523    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.653124    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.663162    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:18.663839    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:18.664292    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-168.ec2.internal.18c009cf5bf76c8d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-168.ec2.internal,UID:ip-10-0-143-168.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-168.ec2.internal,},FirstTimestamp:2026-07-07 14:57:18.644538509 +0000 UTC m=+0.377150819,LastTimestamp:2026-07-07 14:57:18.644538509 +0000 UTC m=+0.377150819,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-168.ec2.internal,}"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.669073    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.669101    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:18.669172    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.685206    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.685222    2581 factory.go:55] Registering systemd factory
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.685227    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.685236    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.685233    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 14:57:18.738745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.685383    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.685391    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.685461    2581 factory.go:153] Registering CRI-O factory
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.685475    2581 factory.go:223] Registration of the crio container factory successfully
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.685521    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.685544    2581 factory.go:103] Registering Raw factory
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.685555    2581 manager.go:1196] Started watching for new ooms in manager
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.686401    2581 manager.go:349] Starting recovery of all containers
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.688248    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:18.709458    2581 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-168.ec2.internal\" not found" node="ip-10-0-143-168.ec2.internal"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.715460    2581 manager.go:354] Recovery completed
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.719672    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.721600    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.721630    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.721640    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.722095    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.722107    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.722120    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.722128    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.723336    2581 policy_none.go:49] "None policy: Start"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.723362    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.723373    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.724544    2581 policy_none.go:47] "Start"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.725886    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.725911    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:57:18.741946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.725924    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:57:18.867647 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:18.769690    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:18.867647 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.780861    2581 manager.go:337] "Starting Device Plugin manager"
Jul 07 14:57:18.867647 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:18.780899    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 14:57:18.867647 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.780908    2581 server.go:87] "Starting device plugin registration server"
Jul 07 14:57:18.867647 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.781179    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 14:57:18.867647 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.781192    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 14:57:18.867647 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.781271    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 14:57:18.867647 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.781336    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 14:57:18.867647 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.781363    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 14:57:18.867647 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:18.781920    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 14:57:18.867647 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:18.782078    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:18.974169 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.882159    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:57:18.974169 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.889796    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:57:18.974169 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.889836    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:57:18.974169 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.889848    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:57:18.974169 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.889876    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-168.ec2.internal"
Jul 07 14:57:18.974169 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.920908    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-168.ec2.internal"
Jul 07 14:57:18.974169 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:18.920941    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-168.ec2.internal\": node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:18.974169 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.924825    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 14:57:18.974169 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.926183    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 14:57:18.974169 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.926224    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 14:57:18.974169 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.926252    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 14:57:18.974169 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:18.926295    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 14:57:18.974169 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:18.938551    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:57:18.974169 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:18.967512    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.028622    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal"]
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.028740    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.030705    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.030733    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.030748    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.032077    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.032187    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.032214    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.032810    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.032832    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.032846    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.032895    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.032917    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.032934    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.034204    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.034228    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.035105    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.035133    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.035146    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:19.066154    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-168.ec2.internal\" not found" node="ip-10-0-143-168.ec2.internal"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:19.067804    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:19.080144 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:19.070124    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-168.ec2.internal\" not found" node="ip-10-0-143-168.ec2.internal"
Jul 07 14:57:19.089398 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.088423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b0c42db611143a1c72c6683cae23b97-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal\" (UID: \"5b0c42db611143a1c72c6683cae23b97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 07 14:57:19.089398 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.088488    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b0c42db611143a1c72c6683cae23b97-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal\" (UID: \"5b0c42db611143a1c72c6683cae23b97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 07 14:57:19.089398 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.088517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/648d8ccbe686a4d38db4c29e47b7d68c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-168.ec2.internal\" (UID: \"648d8ccbe686a4d38db4c29e47b7d68c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal"
Jul 07 14:57:19.230024 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:19.168891    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:19.230024 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.189447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b0c42db611143a1c72c6683cae23b97-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal\" (UID: \"5b0c42db611143a1c72c6683cae23b97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 07 14:57:19.230024 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.189475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b0c42db611143a1c72c6683cae23b97-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal\" (UID: \"5b0c42db611143a1c72c6683cae23b97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 07 14:57:19.230024 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.189497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/648d8ccbe686a4d38db4c29e47b7d68c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-168.ec2.internal\" (UID: \"648d8ccbe686a4d38db4c29e47b7d68c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal"
Jul 07 14:57:19.230024 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.189590    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b0c42db611143a1c72c6683cae23b97-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal\" (UID: \"5b0c42db611143a1c72c6683cae23b97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 07 14:57:19.230024 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.189650    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b0c42db611143a1c72c6683cae23b97-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal\" (UID: \"5b0c42db611143a1c72c6683cae23b97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 07 14:57:19.230024 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.189714    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/648d8ccbe686a4d38db4c29e47b7d68c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-168.ec2.internal\" (UID: \"648d8ccbe686a4d38db4c29e47b7d68c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal"
Jul 07 14:57:19.329910 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:19.272458    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:19.464992 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.372326    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 07 14:57:19.464992 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:19.373434    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:19.464992 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.374656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal"
Jul 07 14:57:19.596875 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:19.474541    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:19.596875 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.570419    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 14:57:19.596875 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.570615    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 14:57:19.596875 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.570629    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 14:57:19.596875 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:19.574748    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:19.714034 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.664994    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 14:52:19 +0000 UTC" deadline="2027-12-02 22:52:24.32245631 +0000 UTC"
Jul 07 14:57:19.714034 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:19.665026    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12319h55m4.657434438s"
Jul 07 14:57:19.714034 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:19.675533    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:19.806152 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:19.776147    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:19.881325 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:19.876464    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:19.977322 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:19.977298    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:20.071333 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.071303    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:57:20.077439 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:20.077400    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 07 14:57:20.098923 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.098902    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:57:20.106390 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:20.106354    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0c42db611143a1c72c6683cae23b97.slice/crio-09d7363dc2328f7faf44ad62d2387dcf5a4b7b08ace40ebad44d467a67515082 WatchSource:0}: Error finding container 09d7363dc2328f7faf44ad62d2387dcf5a4b7b08ace40ebad44d467a67515082: Status 404 returned error can't find the container with id 09d7363dc2328f7faf44ad62d2387dcf5a4b7b08ace40ebad44d467a67515082
Jul 07 14:57:20.106673 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:20.106653    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648d8ccbe686a4d38db4c29e47b7d68c.slice/crio-81da833ceeb192751fdb81dc31de78a76eea0f5725eacaf24f7f3d3634e6a467 WatchSource:0}: Error finding container 81da833ceeb192751fdb81dc31de78a76eea0f5725eacaf24f7f3d3634e6a467: Status 404 returned error can't find the container with id 81da833ceeb192751fdb81dc31de78a76eea0f5725eacaf24f7f3d3634e6a467
Jul 07 14:57:20.113129 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.113108    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 14:57:20.171341 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.171236    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 07 14:57:20.191781 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.191747    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 14:57:20.192708 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.192691    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal"
Jul 07 14:57:20.203313 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.203291    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 14:57:20.635119 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.635014    2581 apiserver.go:52] "Watching apiserver"
Jul 07 14:57:20.644004 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.643975    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 14:57:20.645515 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.645484    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal","openshift-multus/multus-additional-cni-plugins-wwsc4","kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh","openshift-dns/node-resolver-7fg5g","openshift-multus/multus-n292g","openshift-multus/network-metrics-daemon-bjq65","openshift-network-diagnostics/network-check-target-qk5j7","openshift-network-operator/iptables-alerter-kbdbz","openshift-ovn-kubernetes/ovnkube-node-q5z8m","kube-system/global-pull-secret-syncer-b9d6j","kube-system/konnectivity-agent-kwvxp","openshift-cluster-node-tuning-operator/tuned-gn85z","openshift-image-registry/node-ca-72b7k"]
Jul 07 14:57:20.647803 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.647379    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:20.647803 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:20.647484    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:20.649475 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.649195    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.651406 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.651148    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-72b7k"
Jul 07 14:57:20.653654 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.653469    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 14:57:20.653654 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.653482    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:57:20.653838 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.653727    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.653838 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.653826    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6dnp4\""
Jul 07 14:57:20.654721 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.654276    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 14:57:20.654721 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.654461    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 14:57:20.654721 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.654481    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zqzrl\""
Jul 07 14:57:20.654721 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.654607    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 14:57:20.655553 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.655535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7fg5g"
Jul 07 14:57:20.657174 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.656886    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 14:57:20.657492 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.657408    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.659683 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.659662    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 14:57:20.660054 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.660036    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-66psz\""
Jul 07 14:57:20.660141 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.660051    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 14:57:20.660245 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.660228    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kbdbz"
Jul 07 14:57:20.660300 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.660285    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 14:57:20.660607 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.660507    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vt9mh\""
Jul 07 14:57:20.660776 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.660742    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 14:57:20.660961 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.660945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:20.661043 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:20.661014    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:20.661583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.661246    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 14:57:20.661583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.661264    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-t6jjm\""
Jul 07 14:57:20.661583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.661302    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 14:57:20.661583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.661443    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 14:57:20.661583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.661480    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 14:57:20.662354 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.661901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.664996 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.663994    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.664996 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.664864    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 14:57:20.664996 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.664904    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 14:57:20.665217 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.665196    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:20.665329 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:20.665254    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:20.665329 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.665302    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:57:20.666304 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.665878    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 14:57:20.666304 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.666057    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 14:57:20.666304 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.666119    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8hlck\""
Jul 07 14:57:20.666304 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.666128    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fsx5c\""
Jul 07 14:57:20.666858 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.666799    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kwvxp"
Jul 07 14:57:20.671316 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.670470    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 14:57:20.671316 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.670795    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 14:57:20.671316 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.671057    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 14:57:20.671530 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.671317    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 14:57:20.671736 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.671617    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 14:57:20.671736 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.671670    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 14:57:20.671927 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.671773    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p4vxc\""
Jul 07 14:57:20.672590 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.672149    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 14:57:20.675135 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.673184    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 14:57:20.676870 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.676811    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zjjw9\""
Jul 07 14:57:20.677031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.677014    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 14:57:20.685360 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.685324    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 14:57:20.687076 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.687056    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 14:57:20.698335 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-systemd\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.698537 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-run-netns\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.698652 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-run-ovn\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.698652 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-node-log\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.698652 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-sysctl-conf\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.698816 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c0798f1-c292-4322-bc4a-79dda10ef0c9-tmp-dir\") pod \"node-resolver-7fg5g\" (UID: \"4c0798f1-c292-4322-bc4a-79dda10ef0c9\") " pod="openshift-dns/node-resolver-7fg5g"
Jul 07 14:57:20.698816 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b63fcac7-de40-44b0-bdce-81d2e93c8403-cni-binary-copy\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.698816 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-run-multus-certs\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.698816 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ee1eaa63-f9de-41bc-8127-3d5082be4471-env-overrides\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.698816 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698790    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-lib-modules\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.699070 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-host\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.699070 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4c0798f1-c292-4322-bc4a-79dda10ef0c9-hosts-file\") pod \"node-resolver-7fg5g\" (UID: \"4c0798f1-c292-4322-bc4a-79dda10ef0c9\") " pod="openshift-dns/node-resolver-7fg5g"
Jul 07 14:57:20.699070 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-system-cni-dir\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.699070 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-var-lib-cni-bin\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.699070 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698947    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfpxp\" (UniqueName: \"kubernetes.io/projected/ee1eaa63-f9de-41bc-8127-3d5082be4471-kube-api-access-gfpxp\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.699070 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.698990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-kubernetes\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.699070 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb2xn\" (UniqueName: \"kubernetes.io/projected/b9fa874c-f80b-49cc-867c-9d6853c4a95e-kube-api-access-pb2xn\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.699452 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fppb\" (UniqueName: \"kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb\") pod \"network-check-target-qk5j7\" (UID: \"0043d8b0-5817-4b30-950b-0145e8690f3f\") " pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:20.699452 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699111    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:20.699452 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b63fcac7-de40-44b0-bdce-81d2e93c8403-multus-daemon-config\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.699452 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9fa874c-f80b-49cc-867c-9d6853c4a95e-tmp\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.699452 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699187    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13881bc5-ab4d-4258-9e74-195665decbf9-serviceca\") pod \"node-ca-72b7k\" (UID: \"13881bc5-ab4d-4258-9e74-195665decbf9\") " pod="openshift-image-registry/node-ca-72b7k"
Jul 07 14:57:20.699452 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wxkt\" (UniqueName: \"kubernetes.io/projected/bf38e90d-4231-43c1-87f2-d63570a3c3e1-kube-api-access-2wxkt\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.699452 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/386c6191-61a2-412c-9331-152a3d94baf9-kubelet-config\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:20.699452 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-hostroot\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.699452 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-systemd-units\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.699452 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-etc-openvswitch\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.699452 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ee1eaa63-f9de-41bc-8127-3d5082be4471-ovnkube-script-lib\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.699452 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfp6p\" (UniqueName: \"kubernetes.io/projected/13881bc5-ab4d-4258-9e74-195665decbf9-kube-api-access-xfp6p\") pod \"node-ca-72b7k\" (UID: \"13881bc5-ab4d-4258-9e74-195665decbf9\") " pod="openshift-image-registry/node-ca-72b7k"
Jul 07 14:57:20.699849 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-sys-fs\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.699849 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-multus-conf-dir\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.699849 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-run-ovn-kubernetes\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.699849 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-registration-dir\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.699849 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699644    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.699849 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-multus-cni-dir\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.699849 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-run-netns\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.699849 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-etc-kubernetes\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.699849 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwkqj\" (UniqueName: \"kubernetes.io/projected/b63fcac7-de40-44b0-bdce-81d2e93c8403-kube-api-access-nwkqj\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.699849 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdw8z\" (UniqueName: \"kubernetes.io/projected/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-kube-api-access-sdw8z\") pod \"network-metrics-daemon-bjq65\" (UID: \"2c77f4a4-6fbf-4ca3-8291-a48491c2f423\") " pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:20.699849 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-sysctl-d\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.700168 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13881bc5-ab4d-4258-9e74-195665decbf9-host\") pod \"node-ca-72b7k\" (UID: \"13881bc5-ab4d-4258-9e74-195665decbf9\") " pod="openshift-image-registry/node-ca-72b7k"
Jul 07 14:57:20.700168 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-run-k8s-cni-cncf-io\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.700168 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-cni-bin\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.700168 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-cni-netd\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.700168 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.699992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.700168 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-device-dir\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.700168 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqxkz\" (UniqueName: \"kubernetes.io/projected/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-kube-api-access-kqxkz\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.700168 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-os-release\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.700168 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-run-systemd\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.700168 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-sysconfig\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.700530 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-var-lib-kubelet\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.700530 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-tuned\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.700530 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.700530 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-var-lib-kubelet\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.700530 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-slash\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.700530 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700308    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-run-openvswitch\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.700530 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee1eaa63-f9de-41bc-8127-3d5082be4471-ovn-node-metrics-cert\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.700530 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-socket-dir\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.700530 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-cnibin\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.700530 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-kubelet\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.700530 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-modprobe-d\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.701089 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbn2p\" (UniqueName: \"kubernetes.io/projected/4c0798f1-c292-4322-bc4a-79dda10ef0c9-kube-api-access-jbn2p\") pod \"node-resolver-7fg5g\" (UID: \"4c0798f1-c292-4322-bc4a-79dda10ef0c9\") " pod="openshift-dns/node-resolver-7fg5g"
Jul 07 14:57:20.701089 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/213b89a8-eaa5-45bb-a245-20d21e692c38-iptables-alerter-script\") pod \"iptables-alerter-kbdbz\" (UID: \"213b89a8-eaa5-45bb-a245-20d21e692c38\") " pod="openshift-network-operator/iptables-alerter-kbdbz"
Jul 07 14:57:20.701089 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6j2j\" (UniqueName: \"kubernetes.io/projected/213b89a8-eaa5-45bb-a245-20d21e692c38-kube-api-access-h6j2j\") pod \"iptables-alerter-kbdbz\" (UID: \"213b89a8-eaa5-45bb-a245-20d21e692c38\") " pod="openshift-network-operator/iptables-alerter-kbdbz"
Jul 07 14:57:20.701089 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ee1eaa63-f9de-41bc-8127-3d5082be4471-ovnkube-config\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.701089 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700797    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs\") pod \"network-metrics-daemon-bjq65\" (UID: \"2c77f4a4-6fbf-4ca3-8291-a48491c2f423\") " pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:20.701089 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-cni-binary-copy\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.701089 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.701089 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.700917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/386c6191-61a2-412c-9331-152a3d94baf9-dbus\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:20.701089 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.701034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-system-cni-dir\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.701528 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.701277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-cnibin\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.701528 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.701305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/213b89a8-eaa5-45bb-a245-20d21e692c38-host-slash\") pod \"iptables-alerter-kbdbz\" (UID: \"213b89a8-eaa5-45bb-a245-20d21e692c38\") " pod="openshift-network-operator/iptables-alerter-kbdbz"
Jul 07 14:57:20.701528 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.701341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-var-lib-openvswitch\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.701528 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.701414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-var-lib-cni-multus\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.701528 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.701451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-log-socket\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.701528 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.701478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2b9620a4-5364-4242-9b08-e05f0989e587-konnectivity-ca\") pod \"konnectivity-agent-kwvxp\" (UID: \"2b9620a4-5364-4242-9b08-e05f0989e587\") " pod="kube-system/konnectivity-agent-kwvxp"
Jul 07 14:57:20.701528 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.701507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-sys\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.701874 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.701678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-etc-selinux\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.701874 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.701780    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-os-release\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.701874 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.701807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-run\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.701874 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.701833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-multus-socket-dir-parent\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.701874 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.701861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2b9620a4-5364-4242-9b08-e05f0989e587-agent-certs\") pod \"konnectivity-agent-kwvxp\" (UID: \"2b9620a4-5364-4242-9b08-e05f0989e587\") " pod="kube-system/konnectivity-agent-kwvxp"
Jul 07 14:57:20.702118 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.701886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.707439 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.707382    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rtqlp"
Jul 07 14:57:20.716389 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.716341    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rtqlp"
Jul 07 14:57:20.802374 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.802582 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-multus-cni-dir\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.802582 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-run-netns\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.802582 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-etc-kubernetes\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.802582 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802467    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.802582 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwkqj\" (UniqueName: \"kubernetes.io/projected/b63fcac7-de40-44b0-bdce-81d2e93c8403-kube-api-access-nwkqj\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.802582 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdw8z\" (UniqueName: \"kubernetes.io/projected/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-kube-api-access-sdw8z\") pod \"network-metrics-daemon-bjq65\" (UID: \"2c77f4a4-6fbf-4ca3-8291-a48491c2f423\") " pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:20.802904 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802590    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-sysctl-d\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.802904 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13881bc5-ab4d-4258-9e74-195665decbf9-host\") pod \"node-ca-72b7k\" (UID: \"13881bc5-ab4d-4258-9e74-195665decbf9\") " pod="openshift-image-registry/node-ca-72b7k"
Jul 07 14:57:20.802904 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-run-k8s-cni-cncf-io\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.802904 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-cni-bin\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.802904 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802708    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-cni-netd\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.802904 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802711    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-run-netns\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.802904 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802734    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-etc-kubernetes\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.802904 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.802904 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802793    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13881bc5-ab4d-4258-9e74-195665decbf9-host\") pod \"node-ca-72b7k\" (UID: \"13881bc5-ab4d-4258-9e74-195665decbf9\") " pod="openshift-image-registry/node-ca-72b7k"
Jul 07 14:57:20.802904 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-device-dir\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.802904 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802872    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-device-dir\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802920    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-run-k8s-cni-cncf-io\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802923    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-multus-cni-dir\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802924    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-sysctl-d\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802972    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-cni-bin\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.802972    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803005    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-cni-netd\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqxkz\" (UniqueName: \"kubernetes.io/projected/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-kube-api-access-kqxkz\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-os-release\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-run-systemd\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803142    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-run-systemd\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803165    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-os-release\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-sysconfig\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-var-lib-kubelet\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-tuned\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803388    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-sysconfig\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803404    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-var-lib-kubelet\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.803514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-var-lib-kubelet\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-slash\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-run-openvswitch\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee1eaa63-f9de-41bc-8127-3d5082be4471-ovn-node-metrics-cert\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-socket-dir\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-cnibin\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-kubelet\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-modprobe-d\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803645    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-var-lib-kubelet\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbn2p\" (UniqueName: \"kubernetes.io/projected/4c0798f1-c292-4322-bc4a-79dda10ef0c9-kube-api-access-jbn2p\") pod \"node-resolver-7fg5g\" (UID: \"4c0798f1-c292-4322-bc4a-79dda10ef0c9\") " pod="openshift-dns/node-resolver-7fg5g"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/213b89a8-eaa5-45bb-a245-20d21e692c38-iptables-alerter-script\") pod \"iptables-alerter-kbdbz\" (UID: \"213b89a8-eaa5-45bb-a245-20d21e692c38\") " pod="openshift-network-operator/iptables-alerter-kbdbz"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6j2j\" (UniqueName: \"kubernetes.io/projected/213b89a8-eaa5-45bb-a245-20d21e692c38-kube-api-access-h6j2j\") pod \"iptables-alerter-kbdbz\" (UID: \"213b89a8-eaa5-45bb-a245-20d21e692c38\") " pod="openshift-network-operator/iptables-alerter-kbdbz"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ee1eaa63-f9de-41bc-8127-3d5082be4471-ovnkube-config\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs\") pod \"network-metrics-daemon-bjq65\" (UID: \"2c77f4a4-6fbf-4ca3-8291-a48491c2f423\") " pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803944    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-cnibin\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-cni-binary-copy\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803952    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 14:57:20.804338 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803991    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.804026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/386c6191-61a2-412c-9331-152a3d94baf9-dbus\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.804023    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-socket-dir\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803630    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-run-openvswitch\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803964    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-kubelet\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.804161    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.804642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-system-cni-dir\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.804715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-cnibin\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.804053    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-modprobe-d\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.804769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/213b89a8-eaa5-45bb-a245-20d21e692c38-host-slash\") pod \"iptables-alerter-kbdbz\" (UID: \"213b89a8-eaa5-45bb-a245-20d21e692c38\") " pod="openshift-network-operator/iptables-alerter-kbdbz"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.804802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-var-lib-openvswitch\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.804853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-var-lib-cni-multus\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.803586    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-slash\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.804886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-log-socket\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.804950    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-log-socket\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.804951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2b9620a4-5364-4242-9b08-e05f0989e587-konnectivity-ca\") pod \"konnectivity-agent-kwvxp\" (UID: \"2b9620a4-5364-4242-9b08-e05f0989e587\") " pod="kube-system/konnectivity-agent-kwvxp"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-sys\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805022    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-cnibin\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.805218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805026    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ee1eaa63-f9de-41bc-8127-3d5082be4471-ovnkube-config\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-etc-selinux\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-os-release\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805083    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-run\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805090    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-multus-socket-dir-parent\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2b9620a4-5364-4242-9b08-e05f0989e587-agent-certs\") pod \"konnectivity-agent-kwvxp\" (UID: \"2b9620a4-5364-4242-9b08-e05f0989e587\") " pod="kube-system/konnectivity-agent-kwvxp"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805199    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-systemd\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805186    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-system-cni-dir\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-run-netns\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-run-ovn\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805255    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-sys\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-node-log\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805281    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/386c6191-61a2-412c-9331-152a3d94baf9-dbus\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-sysctl-conf\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c0798f1-c292-4322-bc4a-79dda10ef0c9-tmp-dir\") pod \"node-resolver-7fg5g\" (UID: \"4c0798f1-c292-4322-bc4a-79dda10ef0c9\") " pod="openshift-dns/node-resolver-7fg5g"
Jul 07 14:57:20.806175 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805335    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-var-lib-openvswitch\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b63fcac7-de40-44b0-bdce-81d2e93c8403-cni-binary-copy\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805385    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/213b89a8-eaa5-45bb-a245-20d21e692c38-host-slash\") pod \"iptables-alerter-kbdbz\" (UID: \"213b89a8-eaa5-45bb-a245-20d21e692c38\") " pod="openshift-network-operator/iptables-alerter-kbdbz"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-run-multus-certs\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ee1eaa63-f9de-41bc-8127-3d5082be4471-env-overrides\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-lib-modules\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805579    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-run-multus-certs\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805637    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2b9620a4-5364-4242-9b08-e05f0989e587-konnectivity-ca\") pod \"konnectivity-agent-kwvxp\" (UID: \"2b9620a4-5364-4242-9b08-e05f0989e587\") " pod="kube-system/konnectivity-agent-kwvxp"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805692    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-lib-modules\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805701    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-var-lib-cni-multus\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-host\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4c0798f1-c292-4322-bc4a-79dda10ef0c9-hosts-file\") pod \"node-resolver-7fg5g\" (UID: \"4c0798f1-c292-4322-bc4a-79dda10ef0c9\") " pod="openshift-dns/node-resolver-7fg5g"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805775    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-systemd\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-system-cni-dir\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-var-lib-cni-bin\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805847    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-os-release\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfpxp\" (UniqueName: \"kubernetes.io/projected/ee1eaa63-f9de-41bc-8127-3d5082be4471-kube-api-access-gfpxp\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-kubernetes\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.807048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805911    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-multus-socket-dir-parent\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb2xn\" (UniqueName: \"kubernetes.io/projected/b9fa874c-f80b-49cc-867c-9d6853c4a95e-kube-api-access-pb2xn\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fppb\" (UniqueName: \"kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb\") pod \"network-check-target-qk5j7\" (UID: \"0043d8b0-5817-4b30-950b-0145e8690f3f\") " pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805957    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b63fcac7-de40-44b0-bdce-81d2e93c8403-cni-binary-copy\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.805209    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-etc-selinux\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806023    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b63fcac7-de40-44b0-bdce-81d2e93c8403-multus-daemon-config\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806029    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ee1eaa63-f9de-41bc-8127-3d5082be4471-env-overrides\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:20.805129    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806098    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-host\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806099    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-node-log\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806099    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-host-var-lib-cni-bin\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9fa874c-f80b-49cc-867c-9d6853c4a95e-tmp\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806153    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-run-ovn\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13881bc5-ab4d-4258-9e74-195665decbf9-serviceca\") pod \"node-ca-72b7k\" (UID: \"13881bc5-ab4d-4258-9e74-195665decbf9\") " pod="openshift-image-registry/node-ca-72b7k"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806196    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-kubernetes\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wxkt\" (UniqueName: \"kubernetes.io/projected/bf38e90d-4231-43c1-87f2-d63570a3c3e1-kube-api-access-2wxkt\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/386c6191-61a2-412c-9331-152a3d94baf9-kubelet-config\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:20.807883 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-hostroot\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806279    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-sysctl-conf\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806295    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-systemd-units\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806140    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-system-cni-dir\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806341    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4c0798f1-c292-4322-bc4a-79dda10ef0c9-hosts-file\") pod \"node-resolver-7fg5g\" (UID: \"4c0798f1-c292-4322-bc4a-79dda10ef0c9\") " pod="openshift-dns/node-resolver-7fg5g"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806402    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9fa874c-f80b-49cc-867c-9d6853c4a95e-run\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806431    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-systemd-units\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-etc-openvswitch\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806471    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/386c6191-61a2-412c-9331-152a3d94baf9-kubelet-config\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806405    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-hostroot\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ee1eaa63-f9de-41bc-8127-3d5082be4471-ovnkube-script-lib\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806515    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c0798f1-c292-4322-bc4a-79dda10ef0c9-tmp-dir\") pod \"node-resolver-7fg5g\" (UID: \"4c0798f1-c292-4322-bc4a-79dda10ef0c9\") " pod="openshift-dns/node-resolver-7fg5g"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:20.806567    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfp6p\" (UniqueName: \"kubernetes.io/projected/13881bc5-ab4d-4258-9e74-195665decbf9-kube-api-access-xfp6p\") pod \"node-ca-72b7k\" (UID: \"13881bc5-ab4d-4258-9e74-195665decbf9\") " pod="openshift-image-registry/node-ca-72b7k"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-sys-fs\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-multus-conf-dir\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806708    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-run-ovn-kubernetes\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.809171 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-registration-dir\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.810134 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806801    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-run-netns\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.810134 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806871    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-sys-fs\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.810134 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806907    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13881bc5-ab4d-4258-9e74-195665decbf9-serviceca\") pod \"node-ca-72b7k\" (UID: \"13881bc5-ab4d-4258-9e74-195665decbf9\") " pod="openshift-image-registry/node-ca-72b7k"
Jul 07 14:57:20.810134 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806914    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-host-run-ovn-kubernetes\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.810134 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806927    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b63fcac7-de40-44b0-bdce-81d2e93c8403-multus-conf-dir\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.810134 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806934    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf38e90d-4231-43c1-87f2-d63570a3c3e1-registration-dir\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.810134 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.806961    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee1eaa63-f9de-41bc-8127-3d5082be4471-etc-openvswitch\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.810134 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:20.807006    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs podName:2c77f4a4-6fbf-4ca3-8291-a48491c2f423 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:21.306983533 +0000 UTC m=+3.039595853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs") pod "network-metrics-daemon-bjq65" (UID: "2c77f4a4-6fbf-4ca3-8291-a48491c2f423") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:20.810134 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.807124    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ee1eaa63-f9de-41bc-8127-3d5082be4471-ovnkube-script-lib\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.810134 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.807561    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b63fcac7-de40-44b0-bdce-81d2e93c8403-multus-daemon-config\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.810134 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:20.807645    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret podName:386c6191-61a2-412c-9331-152a3d94baf9 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:21.307626819 +0000 UTC m=+3.040239145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret") pod "global-pull-secret-syncer-b9d6j" (UID: "386c6191-61a2-412c-9331-152a3d94baf9") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:20.810134 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.807869    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-cni-binary-copy\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.810134 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.808456    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/213b89a8-eaa5-45bb-a245-20d21e692c38-iptables-alerter-script\") pod \"iptables-alerter-kbdbz\" (UID: \"213b89a8-eaa5-45bb-a245-20d21e692c38\") " pod="openshift-network-operator/iptables-alerter-kbdbz"
Jul 07 14:57:20.811006 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.810935    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee1eaa63-f9de-41bc-8127-3d5082be4471-ovn-node-metrics-cert\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.811150 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.811027    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9fa874c-f80b-49cc-867c-9d6853c4a95e-etc-tuned\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.811150 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.811071    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9fa874c-f80b-49cc-867c-9d6853c4a95e-tmp\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.811318 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.811290    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2b9620a4-5364-4242-9b08-e05f0989e587-agent-certs\") pod \"konnectivity-agent-kwvxp\" (UID: \"2b9620a4-5364-4242-9b08-e05f0989e587\") " pod="kube-system/konnectivity-agent-kwvxp"
Jul 07 14:57:20.823374 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.822797    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:57:20.823992 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.823971    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbn2p\" (UniqueName: \"kubernetes.io/projected/4c0798f1-c292-4322-bc4a-79dda10ef0c9-kube-api-access-jbn2p\") pod \"node-resolver-7fg5g\" (UID: \"4c0798f1-c292-4322-bc4a-79dda10ef0c9\") " pod="openshift-dns/node-resolver-7fg5g"
Jul 07 14:57:20.824753 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.824702    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6j2j\" (UniqueName: \"kubernetes.io/projected/213b89a8-eaa5-45bb-a245-20d21e692c38-kube-api-access-h6j2j\") pod \"iptables-alerter-kbdbz\" (UID: \"213b89a8-eaa5-45bb-a245-20d21e692c38\") " pod="openshift-network-operator/iptables-alerter-kbdbz"
Jul 07 14:57:20.825007 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.824980    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdw8z\" (UniqueName: \"kubernetes.io/projected/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-kube-api-access-sdw8z\") pod \"network-metrics-daemon-bjq65\" (UID: \"2c77f4a4-6fbf-4ca3-8291-a48491c2f423\") " pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:20.826593 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.826545    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqxkz\" (UniqueName: \"kubernetes.io/projected/ce04ed69-ff4c-438a-9261-9f3c66fc6c09-kube-api-access-kqxkz\") pod \"multus-additional-cni-plugins-wwsc4\" (UID: \"ce04ed69-ff4c-438a-9261-9f3c66fc6c09\") " pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:20.826703 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.826653    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwkqj\" (UniqueName: \"kubernetes.io/projected/b63fcac7-de40-44b0-bdce-81d2e93c8403-kube-api-access-nwkqj\") pod \"multus-n292g\" (UID: \"b63fcac7-de40-44b0-bdce-81d2e93c8403\") " pod="openshift-multus/multus-n292g"
Jul 07 14:57:20.827416 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:20.827171    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:57:20.827416 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:20.827208    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:57:20.827416 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:20.827252    2581 projected.go:196] Error preparing data for projected volume kube-api-access-7fppb for pod openshift-network-diagnostics/network-check-target-qk5j7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:20.827416 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:20.827320    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb podName:0043d8b0-5817-4b30-950b-0145e8690f3f nodeName:}" failed. No retries permitted until 2026-07-07 14:57:21.327296093 +0000 UTC m=+3.059908414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7fppb" (UniqueName: "kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb") pod "network-check-target-qk5j7" (UID: "0043d8b0-5817-4b30-950b-0145e8690f3f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:20.830547 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.829685    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfpxp\" (UniqueName: \"kubernetes.io/projected/ee1eaa63-f9de-41bc-8127-3d5082be4471-kube-api-access-gfpxp\") pod \"ovnkube-node-q5z8m\" (UID: \"ee1eaa63-f9de-41bc-8127-3d5082be4471\") " pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:20.830547 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.830022    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wxkt\" (UniqueName: \"kubernetes.io/projected/bf38e90d-4231-43c1-87f2-d63570a3c3e1-kube-api-access-2wxkt\") pod \"aws-ebs-csi-driver-node-7qrrh\" (UID: \"bf38e90d-4231-43c1-87f2-d63570a3c3e1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.832690 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.831399    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb2xn\" (UniqueName: \"kubernetes.io/projected/b9fa874c-f80b-49cc-867c-9d6853c4a95e-kube-api-access-pb2xn\") pod \"tuned-gn85z\" (UID: \"b9fa874c-f80b-49cc-867c-9d6853c4a95e\") " pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.832690 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.831595    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfp6p\" (UniqueName: \"kubernetes.io/projected/13881bc5-ab4d-4258-9e74-195665decbf9-kube-api-access-xfp6p\") pod \"node-ca-72b7k\" (UID: \"13881bc5-ab4d-4258-9e74-195665decbf9\") " pod="openshift-image-registry/node-ca-72b7k"
Jul 07 14:57:20.941611 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.941512    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal" event={"ID":"5b0c42db611143a1c72c6683cae23b97","Type":"ContainerStarted","Data":"09d7363dc2328f7faf44ad62d2387dcf5a4b7b08ace40ebad44d467a67515082"}
Jul 07 14:57:20.942986 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.942928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal" event={"ID":"648d8ccbe686a4d38db4c29e47b7d68c","Type":"ContainerStarted","Data":"81da833ceeb192751fdb81dc31de78a76eea0f5725eacaf24f7f3d3634e6a467"}
Jul 07 14:57:20.966204 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.966169    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gn85z"
Jul 07 14:57:20.977533 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:20.977463    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fa874c_f80b_49cc_867c_9d6853c4a95e.slice/crio-49adbca06b96f0202a5dc9591f3ce09bf95989ae3e57b1b37b946febcb747749 WatchSource:0}: Error finding container 49adbca06b96f0202a5dc9591f3ce09bf95989ae3e57b1b37b946febcb747749: Status 404 returned error can't find the container with id 49adbca06b96f0202a5dc9591f3ce09bf95989ae3e57b1b37b946febcb747749
Jul 07 14:57:20.980515 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.980459    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-72b7k"
Jul 07 14:57:20.992341 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.992157    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh"
Jul 07 14:57:20.992469 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:20.992444    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13881bc5_ab4d_4258_9e74_195665decbf9.slice/crio-e7dbba4c5f302c605ebb9c3e77d2c1dbbcd1875fb15e527c39a4b55619d05a52 WatchSource:0}: Error finding container e7dbba4c5f302c605ebb9c3e77d2c1dbbcd1875fb15e527c39a4b55619d05a52: Status 404 returned error can't find the container with id e7dbba4c5f302c605ebb9c3e77d2c1dbbcd1875fb15e527c39a4b55619d05a52
Jul 07 14:57:20.998016 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:20.997989    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7fg5g"
Jul 07 14:57:21.002547 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:21.002519    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf38e90d_4231_43c1_87f2_d63570a3c3e1.slice/crio-e3afa5d05947a90cf6c18240fc6b7f227e41b32c289d97da2179e67869daf5c6 WatchSource:0}: Error finding container e3afa5d05947a90cf6c18240fc6b7f227e41b32c289d97da2179e67869daf5c6: Status 404 returned error can't find the container with id e3afa5d05947a90cf6c18240fc6b7f227e41b32c289d97da2179e67869daf5c6
Jul 07 14:57:21.007422 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.007399    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n292g"
Jul 07 14:57:21.009084 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:21.009054    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0798f1_c292_4322_bc4a_79dda10ef0c9.slice/crio-a6d3aa24c3229e79328120ba7ef52684f5773555c29b567cf8bea28772fddca4 WatchSource:0}: Error finding container a6d3aa24c3229e79328120ba7ef52684f5773555c29b567cf8bea28772fddca4: Status 404 returned error can't find the container with id a6d3aa24c3229e79328120ba7ef52684f5773555c29b567cf8bea28772fddca4
Jul 07 14:57:21.017011 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.016984    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kbdbz"
Jul 07 14:57:21.017385 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:21.017336    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63fcac7_de40_44b0_bdce_81d2e93c8403.slice/crio-9f83c8b37b6ce998b30b856f8c65496b47625fc45fd517678187f4a4272b8bb9 WatchSource:0}: Error finding container 9f83c8b37b6ce998b30b856f8c65496b47625fc45fd517678187f4a4272b8bb9: Status 404 returned error can't find the container with id 9f83c8b37b6ce998b30b856f8c65496b47625fc45fd517678187f4a4272b8bb9
Jul 07 14:57:21.025899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.025873    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wwsc4"
Jul 07 14:57:21.027546 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:21.027520    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213b89a8_eaa5_45bb_a245_20d21e692c38.slice/crio-4961ff8505cad9fee27bfbba8b22267b4068d27eae42251b7fd5b9d0e39cfa72 WatchSource:0}: Error finding container 4961ff8505cad9fee27bfbba8b22267b4068d27eae42251b7fd5b9d0e39cfa72: Status 404 returned error can't find the container with id 4961ff8505cad9fee27bfbba8b22267b4068d27eae42251b7fd5b9d0e39cfa72
Jul 07 14:57:21.034867 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.034662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:21.037712 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:21.037681    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce04ed69_ff4c_438a_9261_9f3c66fc6c09.slice/crio-2f20d1dfa467e7309d4af86911dc470eb120ae1a3c444736929fd745cefe91e7 WatchSource:0}: Error finding container 2f20d1dfa467e7309d4af86911dc470eb120ae1a3c444736929fd745cefe91e7: Status 404 returned error can't find the container with id 2f20d1dfa467e7309d4af86911dc470eb120ae1a3c444736929fd745cefe91e7
Jul 07 14:57:21.041419 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.041398    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kwvxp"
Jul 07 14:57:21.045587 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:21.045559    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1eaa63_f9de_41bc_8127_3d5082be4471.slice/crio-c27de58a32cd72a37cd2f1ca4206655ee1d216a0268fc583d3e669439049111f WatchSource:0}: Error finding container c27de58a32cd72a37cd2f1ca4206655ee1d216a0268fc583d3e669439049111f: Status 404 returned error can't find the container with id c27de58a32cd72a37cd2f1ca4206655ee1d216a0268fc583d3e669439049111f
Jul 07 14:57:21.048393 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.048346    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:57:21.310773 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.310687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:21.310773 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.310740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs\") pod \"network-metrics-daemon-bjq65\" (UID: \"2c77f4a4-6fbf-4ca3-8291-a48491c2f423\") " pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:21.310990 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:21.310869    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:21.310990 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:21.310873    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:21.310990 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:21.310937    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs podName:2c77f4a4-6fbf-4ca3-8291-a48491c2f423 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:22.310915364 +0000 UTC m=+4.043527674 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs") pod "network-metrics-daemon-bjq65" (UID: "2c77f4a4-6fbf-4ca3-8291-a48491c2f423") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:21.310990 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:21.310955    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret podName:386c6191-61a2-412c-9331-152a3d94baf9 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:22.310947877 +0000 UTC m=+4.043560180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret") pod "global-pull-secret-syncer-b9d6j" (UID: "386c6191-61a2-412c-9331-152a3d94baf9") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:21.411299 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.411175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fppb\" (UniqueName: \"kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb\") pod \"network-check-target-qk5j7\" (UID: \"0043d8b0-5817-4b30-950b-0145e8690f3f\") " pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:21.411484 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:21.411432    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:57:21.411484 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:21.411461    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:57:21.411484 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:21.411477    2581 projected.go:196] Error preparing data for projected volume kube-api-access-7fppb for pod openshift-network-diagnostics/network-check-target-qk5j7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:21.411636 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:21.411569    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb podName:0043d8b0-5817-4b30-950b-0145e8690f3f nodeName:}" failed. No retries permitted until 2026-07-07 14:57:22.411542402 +0000 UTC m=+4.144154786 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7fppb" (UniqueName: "kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb") pod "network-check-target-qk5j7" (UID: "0043d8b0-5817-4b30-950b-0145e8690f3f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:21.719527 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.719418    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:52:21 +0000 UTC" deadline="2028-02-01 14:57:16.390366392 +0000 UTC"
Jul 07 14:57:21.719527 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.719459    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13775h59m54.670912981s"
Jul 07 14:57:21.927445 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.927402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:21.927633 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:21.927540    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:21.927699 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.927646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:21.927699 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.927670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:21.927852 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:21.927777    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:21.927913 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:21.927877    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:21.946684 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.946646    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m" event={"ID":"ee1eaa63-f9de-41bc-8127-3d5082be4471","Type":"ContainerStarted","Data":"c27de58a32cd72a37cd2f1ca4206655ee1d216a0268fc583d3e669439049111f"}
Jul 07 14:57:21.947947 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.947916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwsc4" event={"ID":"ce04ed69-ff4c-438a-9261-9f3c66fc6c09","Type":"ContainerStarted","Data":"2f20d1dfa467e7309d4af86911dc470eb120ae1a3c444736929fd745cefe91e7"}
Jul 07 14:57:21.949465 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.949436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kbdbz" event={"ID":"213b89a8-eaa5-45bb-a245-20d21e692c38","Type":"ContainerStarted","Data":"4961ff8505cad9fee27bfbba8b22267b4068d27eae42251b7fd5b9d0e39cfa72"}
Jul 07 14:57:21.951483 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.951255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7fg5g" event={"ID":"4c0798f1-c292-4322-bc4a-79dda10ef0c9","Type":"ContainerStarted","Data":"a6d3aa24c3229e79328120ba7ef52684f5773555c29b567cf8bea28772fddca4"}
Jul 07 14:57:21.952850 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.952826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-72b7k" event={"ID":"13881bc5-ab4d-4258-9e74-195665decbf9","Type":"ContainerStarted","Data":"e7dbba4c5f302c605ebb9c3e77d2c1dbbcd1875fb15e527c39a4b55619d05a52"}
Jul 07 14:57:21.954657 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.954631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n292g" event={"ID":"b63fcac7-de40-44b0-bdce-81d2e93c8403","Type":"ContainerStarted","Data":"9f83c8b37b6ce998b30b856f8c65496b47625fc45fd517678187f4a4272b8bb9"}
Jul 07 14:57:21.956299 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.956276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh" event={"ID":"bf38e90d-4231-43c1-87f2-d63570a3c3e1","Type":"ContainerStarted","Data":"e3afa5d05947a90cf6c18240fc6b7f227e41b32c289d97da2179e67869daf5c6"}
Jul 07 14:57:21.958427 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:21.958357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gn85z" event={"ID":"b9fa874c-f80b-49cc-867c-9d6853c4a95e","Type":"ContainerStarted","Data":"49adbca06b96f0202a5dc9591f3ce09bf95989ae3e57b1b37b946febcb747749"}
Jul 07 14:57:22.318777 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:22.318737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs\") pod \"network-metrics-daemon-bjq65\" (UID: \"2c77f4a4-6fbf-4ca3-8291-a48491c2f423\") " pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:22.318935 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:22.318870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:22.319135 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:22.319038    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:22.319135 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:22.319086    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:22.319135 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:22.319132    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret podName:386c6191-61a2-412c-9331-152a3d94baf9 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:24.319109973 +0000 UTC m=+6.051722287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret") pod "global-pull-secret-syncer-b9d6j" (UID: "386c6191-61a2-412c-9331-152a3d94baf9") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:22.319349 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:22.319154    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs podName:2c77f4a4-6fbf-4ca3-8291-a48491c2f423 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:24.319143906 +0000 UTC m=+6.051756205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs") pod "network-metrics-daemon-bjq65" (UID: "2c77f4a4-6fbf-4ca3-8291-a48491c2f423") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:22.340071 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:57:22.340021    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9620a4_5364_4242_9b08_e05f0989e587.slice/crio-170aac5905e406bf28ed5c7d04d236ed6b0eb717daaac434881bf3fbd1b5e98b WatchSource:0}: Error finding container 170aac5905e406bf28ed5c7d04d236ed6b0eb717daaac434881bf3fbd1b5e98b: Status 404 returned error can't find the container with id 170aac5905e406bf28ed5c7d04d236ed6b0eb717daaac434881bf3fbd1b5e98b
Jul 07 14:57:22.420648 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:22.419917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fppb\" (UniqueName: \"kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb\") pod \"network-check-target-qk5j7\" (UID: \"0043d8b0-5817-4b30-950b-0145e8690f3f\") " pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:22.420648 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:22.420099    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:57:22.420648 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:22.420119    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:57:22.420648 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:22.420132    2581 projected.go:196] Error preparing data for projected volume kube-api-access-7fppb for pod openshift-network-diagnostics/network-check-target-qk5j7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:22.420648 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:22.420192    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb podName:0043d8b0-5817-4b30-950b-0145e8690f3f nodeName:}" failed. No retries permitted until 2026-07-07 14:57:24.420172786 +0000 UTC m=+6.152785101 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7fppb" (UniqueName: "kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb") pod "network-check-target-qk5j7" (UID: "0043d8b0-5817-4b30-950b-0145e8690f3f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:22.720647 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:22.720177    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:52:21 +0000 UTC" deadline="2028-03-11 07:45:51.093237457 +0000 UTC"
Jul 07 14:57:22.720647 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:22.720469    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14704h48m28.372774783s"
Jul 07 14:57:22.963676 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:22.963615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kwvxp" event={"ID":"2b9620a4-5364-4242-9b08-e05f0989e587","Type":"ContainerStarted","Data":"170aac5905e406bf28ed5c7d04d236ed6b0eb717daaac434881bf3fbd1b5e98b"}
Jul 07 14:57:22.966677 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:22.966639    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal" event={"ID":"648d8ccbe686a4d38db4c29e47b7d68c","Type":"ContainerStarted","Data":"159ee9aecc6ca96f7bf899e697f00764a2d55ccd9bbf7b13b875fb462bd0c8ff"}
Jul 07 14:57:23.926995 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:23.926957    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:23.927583 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:23.927111    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:23.927583 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:23.927543    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:23.927727 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:23.927633    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:23.927727 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:23.927702    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:23.927851 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:23.927779    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:23.974389 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:23.974215    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal" event={"ID":"5b0c42db611143a1c72c6683cae23b97","Type":"ContainerStarted","Data":"6d1d777d2c7453a36fae429697f967101d07b19de9ac73514f26b5b6c30d3245"}
Jul 07 14:57:24.005204 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:24.005116    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal" podStartSLOduration=4.005090011 podStartE2EDuration="4.005090011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:22.989224169 +0000 UTC m=+4.721836492" watchObservedRunningTime="2026-07-07 14:57:24.005090011 +0000 UTC m=+5.737702333"
Jul 07 14:57:24.339014 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:24.338915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:24.339014 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:24.338992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs\") pod \"network-metrics-daemon-bjq65\" (UID: \"2c77f4a4-6fbf-4ca3-8291-a48491c2f423\") " pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:24.339245 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:24.339124    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:24.339245 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:24.339188    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs podName:2c77f4a4-6fbf-4ca3-8291-a48491c2f423 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:28.339167965 +0000 UTC m=+10.071780280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs") pod "network-metrics-daemon-bjq65" (UID: "2c77f4a4-6fbf-4ca3-8291-a48491c2f423") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:24.339646 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:24.339625    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:24.339800 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:24.339687    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret podName:386c6191-61a2-412c-9331-152a3d94baf9 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:28.339670521 +0000 UTC m=+10.072282837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret") pod "global-pull-secret-syncer-b9d6j" (UID: "386c6191-61a2-412c-9331-152a3d94baf9") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:24.439925 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:24.439886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fppb\" (UniqueName: \"kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb\") pod \"network-check-target-qk5j7\" (UID: \"0043d8b0-5817-4b30-950b-0145e8690f3f\") " pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:24.440211 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:24.440114    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:57:24.440211 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:24.440135    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:57:24.440211 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:24.440149    2581 projected.go:196] Error preparing data for projected volume kube-api-access-7fppb for pod openshift-network-diagnostics/network-check-target-qk5j7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:24.440211 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:24.440208    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb podName:0043d8b0-5817-4b30-950b-0145e8690f3f nodeName:}" failed. No retries permitted until 2026-07-07 14:57:28.44018908 +0000 UTC m=+10.172801396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7fppb" (UniqueName: "kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb") pod "network-check-target-qk5j7" (UID: "0043d8b0-5817-4b30-950b-0145e8690f3f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:25.931344 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:25.928686    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:25.931344 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:25.928884    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:25.931344 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:25.929390    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:25.931344 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:25.929536    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:25.931344 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:25.929611    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:25.931344 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:25.929712    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:26.981939 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:26.981829    2581 generic.go:356] "Generic (PLEG): container finished" podID="5b0c42db611143a1c72c6683cae23b97" containerID="6d1d777d2c7453a36fae429697f967101d07b19de9ac73514f26b5b6c30d3245" exitCode=0
Jul 07 14:57:26.981939 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:26.981897    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal" event={"ID":"5b0c42db611143a1c72c6683cae23b97","Type":"ContainerDied","Data":"6d1d777d2c7453a36fae429697f967101d07b19de9ac73514f26b5b6c30d3245"}
Jul 07 14:57:27.926585 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:27.926496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:27.926585 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:27.926528    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:27.926876 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:27.926619    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:27.926936 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:27.926498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:27.927177 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:27.927022    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:27.927177 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:27.927133    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:28.375560 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:28.375006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs\") pod \"network-metrics-daemon-bjq65\" (UID: \"2c77f4a4-6fbf-4ca3-8291-a48491c2f423\") " pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:28.375560 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:28.375105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:28.375560 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:28.375251    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:28.375560 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:28.375328    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret podName:386c6191-61a2-412c-9331-152a3d94baf9 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:36.375304326 +0000 UTC m=+18.107916727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret") pod "global-pull-secret-syncer-b9d6j" (UID: "386c6191-61a2-412c-9331-152a3d94baf9") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:28.375560 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:28.375407    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:28.375560 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:28.375477    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs podName:2c77f4a4-6fbf-4ca3-8291-a48491c2f423 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:36.37546003 +0000 UTC m=+18.108072334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs") pod "network-metrics-daemon-bjq65" (UID: "2c77f4a4-6fbf-4ca3-8291-a48491c2f423") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:28.476118 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:28.476077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fppb\" (UniqueName: \"kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb\") pod \"network-check-target-qk5j7\" (UID: \"0043d8b0-5817-4b30-950b-0145e8690f3f\") " pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:28.476319 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:28.476288    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:57:28.476319 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:28.476318    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:57:28.476462 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:28.476333    2581 projected.go:196] Error preparing data for projected volume kube-api-access-7fppb for pod openshift-network-diagnostics/network-check-target-qk5j7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:28.476462 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:28.476432    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb podName:0043d8b0-5817-4b30-950b-0145e8690f3f nodeName:}" failed. No retries permitted until 2026-07-07 14:57:36.47640911 +0000 UTC m=+18.209021435 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7fppb" (UniqueName: "kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb") pod "network-check-target-qk5j7" (UID: "0043d8b0-5817-4b30-950b-0145e8690f3f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:29.926757 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:29.926725    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:29.927233 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:29.926892    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:29.927233 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:29.927042    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:29.927233 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:29.927154    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:29.927727 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:29.927708    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:29.927901 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:29.927877    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:31.926491 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:31.926459    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:31.926491 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:31.926477    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:31.926999 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:31.926473    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:31.926999 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:31.926591    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:31.926999 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:31.926733    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:31.926999 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:31.926800    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:33.926989 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:33.926953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:33.927453 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:33.926953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:33.927453 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:33.927096    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:33.927453 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:33.927143    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:33.927453 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:33.926953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:33.927453 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:33.927228    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:35.927402 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:35.927363    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:35.927402 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:35.927403    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:35.927934 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:35.927482    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:35.927934 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:35.927372    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:35.927934 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:35.927564    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:35.927934 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:35.927634    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:36.436390 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:36.436339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs\") pod \"network-metrics-daemon-bjq65\" (UID: \"2c77f4a4-6fbf-4ca3-8291-a48491c2f423\") " pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:36.436575 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:36.436425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:36.436575 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:36.436478    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:36.436575 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:36.436533    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:36.436575 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:36.436548    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs podName:2c77f4a4-6fbf-4ca3-8291-a48491c2f423 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:52.436527858 +0000 UTC m=+34.169140159 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs") pod "network-metrics-daemon-bjq65" (UID: "2c77f4a4-6fbf-4ca3-8291-a48491c2f423") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:36.436736 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:36.436580    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret podName:386c6191-61a2-412c-9331-152a3d94baf9 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:52.436565349 +0000 UTC m=+34.169177651 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret") pod "global-pull-secret-syncer-b9d6j" (UID: "386c6191-61a2-412c-9331-152a3d94baf9") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:36.537024 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:36.536993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fppb\" (UniqueName: \"kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb\") pod \"network-check-target-qk5j7\" (UID: \"0043d8b0-5817-4b30-950b-0145e8690f3f\") " pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:36.537190 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:36.537134    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:57:36.537190 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:36.537150    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:57:36.537190 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:36.537160    2581 projected.go:196] Error preparing data for projected volume kube-api-access-7fppb for pod openshift-network-diagnostics/network-check-target-qk5j7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:36.537322 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:36.537215    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb podName:0043d8b0-5817-4b30-950b-0145e8690f3f nodeName:}" failed. No retries permitted until 2026-07-07 14:57:52.53719769 +0000 UTC m=+34.269810005 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7fppb" (UniqueName: "kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb") pod "network-check-target-qk5j7" (UID: "0043d8b0-5817-4b30-950b-0145e8690f3f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:37.927126 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:37.927090    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:37.927552 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:37.927142    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:37.927552 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:37.927090    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:37.927552 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:37.927223    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:37.927552 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:37.927339    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:37.927552 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:37.927430    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:39.926715 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:39.926674    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:39.926715 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:39.926703    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:39.927269 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:39.926692    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:39.927269 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:39.926810    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:39.927269 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:39.926867    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:39.927269 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:39.926950    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:41.927026 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:41.926807    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:41.927965 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:41.926858    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:41.927965 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:41.927127    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:41.927965 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:41.926953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:41.927965 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:41.927279    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:41.927965 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:41.927388    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:42.014010 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.013971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7fg5g" event={"ID":"4c0798f1-c292-4322-bc4a-79dda10ef0c9","Type":"ContainerStarted","Data":"3ea298841b84952a18a851a15dd6ad87e99d0b5dfaf3cb090d44033cea2b89c1"}
Jul 07 14:57:42.015691 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.015659    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-72b7k" event={"ID":"13881bc5-ab4d-4258-9e74-195665decbf9","Type":"ContainerStarted","Data":"937f689921abec22b3aa2db0ed49f03621f33b18c607bd0071a584bc243da766"}
Jul 07 14:57:42.017595 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.017566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal" event={"ID":"5b0c42db611143a1c72c6683cae23b97","Type":"ContainerStarted","Data":"e2eb84b18cd238a15478e147bc437f3f318ee4b449032bae1ed0036901c2c134"}
Jul 07 14:57:42.019190 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.019158    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n292g" event={"ID":"b63fcac7-de40-44b0-bdce-81d2e93c8403","Type":"ContainerStarted","Data":"aca0b7ad1b240705d739504c22fdfd91f68108495896612add2159032ea4d844"}
Jul 07 14:57:42.020851 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.020826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh" event={"ID":"bf38e90d-4231-43c1-87f2-d63570a3c3e1","Type":"ContainerStarted","Data":"ded3e8f23a5dfacb8e83b82012a2f88c75e633d15d2a3a23decc7036cd795070"}
Jul 07 14:57:42.022503 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.022480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gn85z" event={"ID":"b9fa874c-f80b-49cc-867c-9d6853c4a95e","Type":"ContainerStarted","Data":"4eb0c1f7966e942a8b562209f25b51a46419a6929184dc104b9b5b869680805f"}
Jul 07 14:57:42.024005 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.023977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kwvxp" event={"ID":"2b9620a4-5364-4242-9b08-e05f0989e587","Type":"ContainerStarted","Data":"e50cbff2dd6b45d72ec754680a3c804192c13ac1ebe40d5e9122e30fad3c9fde"}
Jul 07 14:57:42.027379 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.027336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m" event={"ID":"ee1eaa63-f9de-41bc-8127-3d5082be4471","Type":"ContainerStarted","Data":"9afb1f72ff21225c1ea45ae10221cd5405280cbc658db6014d1605c98d4af3ca"}
Jul 07 14:57:42.027486 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.027430    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m" event={"ID":"ee1eaa63-f9de-41bc-8127-3d5082be4471","Type":"ContainerStarted","Data":"6706b8aa032542198e3b10ad65e63cce9b609f2b0d39ece46659b05a624db0c4"}
Jul 07 14:57:42.027550 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.027485    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m" event={"ID":"ee1eaa63-f9de-41bc-8127-3d5082be4471","Type":"ContainerStarted","Data":"48734e636b361474a29cd22a6e9af956596097410079b16d4fc6a81316d697ed"}
Jul 07 14:57:42.027550 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.027538    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m" event={"ID":"ee1eaa63-f9de-41bc-8127-3d5082be4471","Type":"ContainerStarted","Data":"dd70befea020ebf18b8d89c200f70387e916b242bd62c87919c3c7bc0910aa82"}
Jul 07 14:57:42.027659 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.027586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m" event={"ID":"ee1eaa63-f9de-41bc-8127-3d5082be4471","Type":"ContainerStarted","Data":"1235a508e4431a2a127f8c8b5fff33338364ab169c7d982c30c09cdae8dc0f8a"}
Jul 07 14:57:42.027659 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.027638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m" event={"ID":"ee1eaa63-f9de-41bc-8127-3d5082be4471","Type":"ContainerStarted","Data":"de337d4e0d7dd1f796ec400caaa02c792675612db55e85d6f274c3fb33a0de87"}
Jul 07 14:57:42.029034 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.029008    2581 generic.go:356] "Generic (PLEG): container finished" podID="ce04ed69-ff4c-438a-9261-9f3c66fc6c09" containerID="d3dee3dd65d51ddccb9c03c1145d3bbe969255cb01f7e33f83414ecfb5cbf441" exitCode=0
Jul 07 14:57:42.029132 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.029043    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwsc4" event={"ID":"ce04ed69-ff4c-438a-9261-9f3c66fc6c09","Type":"ContainerDied","Data":"d3dee3dd65d51ddccb9c03c1145d3bbe969255cb01f7e33f83414ecfb5cbf441"}
Jul 07 14:57:42.032820 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.032784    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7fg5g" podStartSLOduration=3.014503795 podStartE2EDuration="23.032772216s" totalImagesPullingTime="20.018268421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:42.032568842 +0000 UTC m=+23.765181163" watchObservedRunningTime="2026-07-07 14:57:42.032772216 +0000 UTC m=+23.765384535"
Jul 07 14:57:42.054159 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.054105    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal" podStartSLOduration=22.054086618 podStartE2EDuration="22.054086618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:42.053906484 +0000 UTC m=+23.786518805" watchObservedRunningTime="2026-07-07 14:57:42.054086618 +0000 UTC m=+23.786698945"
Jul 07 14:57:42.079061 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.079009    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-72b7k" podStartSLOduration=3.077792325 podStartE2EDuration="23.078993875s" totalImagesPullingTime="20.00120155s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:42.078687924 +0000 UTC m=+23.811300244" watchObservedRunningTime="2026-07-07 14:57:42.078993875 +0000 UTC m=+23.811606194"
Jul 07 14:57:42.131318 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.131264    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n292g" podStartSLOduration=2.9987673790000002 podStartE2EDuration="23.131250914s" totalImagesPullingTime="20.132483535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:42.105061195 +0000 UTC m=+23.837673516" watchObservedRunningTime="2026-07-07 14:57:42.131250914 +0000 UTC m=+23.863863234"
Jul 07 14:57:42.157118 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.157036    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gn85z" podStartSLOduration=3.16807279 podStartE2EDuration="23.157013227s" totalImagesPullingTime="19.988940437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:42.156709348 +0000 UTC m=+23.889321670" watchObservedRunningTime="2026-07-07 14:57:42.157013227 +0000 UTC m=+23.889625547"
Jul 07 14:57:42.181651 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.181554    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kwvxp" podStartSLOduration=4.494456677 podStartE2EDuration="23.181539077s" totalImagesPullingTime="18.6870824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:42.181440733 +0000 UTC m=+23.914053098" watchObservedRunningTime="2026-07-07 14:57:42.181539077 +0000 UTC m=+23.914151396"
Jul 07 14:57:42.601258 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.601231    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 14:57:42.796101 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.795888    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T14:57:42.60125205Z","UUID":"f5820e3e-c016-4bab-b6f0-797bf833f919","Handler":null,"Name":"","Endpoint":""}
Jul 07 14:57:42.797620 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.797595    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 14:57:42.797743 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:42.797630    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 14:57:43.032803 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:43.032769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh" event={"ID":"bf38e90d-4231-43c1-87f2-d63570a3c3e1","Type":"ContainerStarted","Data":"fbdb39bc00ec0ba198fdcc1047d9b8c841f4f82d3f85cdb711a75e980f65b44e"}
Jul 07 14:57:43.034271 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:43.034237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kbdbz" event={"ID":"213b89a8-eaa5-45bb-a245-20d21e692c38","Type":"ContainerStarted","Data":"f40f94a980454cb8f1573703e217e0043ab2a187f4a0f277f6de24dac67adb9c"}
Jul 07 14:57:43.053366 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:43.053243    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kbdbz" podStartSLOduration=4.087592417 podStartE2EDuration="24.053226988s" totalImagesPullingTime="19.965634571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:43.053084353 +0000 UTC m=+24.785696664" watchObservedRunningTime="2026-07-07 14:57:43.053226988 +0000 UTC m=+24.785839286"
Jul 07 14:57:43.927383 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:43.927340    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:43.927383 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:43.927372    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:43.927703 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:43.927337    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:43.927703 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:43.927483    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:43.927703 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:43.927608    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:43.927703 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:43.927697    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:44.038690 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:44.038610    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh" event={"ID":"bf38e90d-4231-43c1-87f2-d63570a3c3e1","Type":"ContainerStarted","Data":"a0788f8e78c17a50470509abfb8532a47726fde2f73132bad51a1f4442e70ee8"}
Jul 07 14:57:44.042652 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:44.042622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m" event={"ID":"ee1eaa63-f9de-41bc-8127-3d5082be4471","Type":"ContainerStarted","Data":"4d4b879d0b5f1667f49c768418214ec4601e6f588907945af9d2d390067990b3"}
Jul 07 14:57:44.063242 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:44.063186    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qrrh" podStartSLOduration=2.767121891 podStartE2EDuration="25.06317089s" totalImagesPullingTime="22.296048999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:57:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:44.06300611 +0000 UTC m=+25.795618431" watchObservedRunningTime="2026-07-07 14:57:44.06317089 +0000 UTC m=+25.795783209"
Jul 07 14:57:44.848321 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:44.848283    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kwvxp"
Jul 07 14:57:44.849040 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:44.849017    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kwvxp"
Jul 07 14:57:45.047149 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:45.047112    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kwvxp"
Jul 07 14:57:45.047716 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:45.047693    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kwvxp"
Jul 07 14:57:45.926793 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:45.926760    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:45.926979 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:45.926762    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:45.926979 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:45.926861    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:45.926979 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:45.926762    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:45.927147 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:45.926962    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:45.927147 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:45.927081    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:47.054502 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:47.054284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m" event={"ID":"ee1eaa63-f9de-41bc-8127-3d5082be4471","Type":"ContainerStarted","Data":"17daa3c23c23f232044998d1a39c33ee07e17d792cb27ece61ac12d5e83c8af5"}
Jul 07 14:57:47.055305 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:47.054672    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:47.055305 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:47.054734    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:47.055305 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:47.054773    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:47.056468 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:47.056440    2581 generic.go:356] "Generic (PLEG): container finished" podID="ce04ed69-ff4c-438a-9261-9f3c66fc6c09" containerID="5ea0bcc755022c6ddb8e2333db44bea97753bab3cc9d9f1c8e943e5eb538c459" exitCode=0
Jul 07 14:57:47.056572 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:47.056520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwsc4" event={"ID":"ce04ed69-ff4c-438a-9261-9f3c66fc6c09","Type":"ContainerDied","Data":"5ea0bcc755022c6ddb8e2333db44bea97753bab3cc9d9f1c8e943e5eb538c459"}
Jul 07 14:57:47.070521 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:47.070501    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:47.073728 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:47.073711    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:57:47.235634 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:47.235576    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m" podStartSLOduration=8.199880062 podStartE2EDuration="28.235559992s" totalImagesPullingTime="20.03567993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:47.235212608 +0000 UTC m=+28.967824929" watchObservedRunningTime="2026-07-07 14:57:47.235559992 +0000 UTC m=+28.968172311"
Jul 07 14:57:47.927179 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:47.927144    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:47.927179 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:47.927160    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:47.927179 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:47.927160    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:47.927515 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:47.927270    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:47.927515 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:47.927300    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:47.927515 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:47.927378    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:49.063198 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:49.063166    2581 generic.go:356] "Generic (PLEG): container finished" podID="ce04ed69-ff4c-438a-9261-9f3c66fc6c09" containerID="e3e5ace06980e585c7caa0cf56f7df35817119a4a80ca87a0a9480bf2cdf40ae" exitCode=0
Jul 07 14:57:49.063614 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:49.063253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwsc4" event={"ID":"ce04ed69-ff4c-438a-9261-9f3c66fc6c09","Type":"ContainerDied","Data":"e3e5ace06980e585c7caa0cf56f7df35817119a4a80ca87a0a9480bf2cdf40ae"}
Jul 07 14:57:49.927136 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:49.927095    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:49.927306 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:49.927097    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:49.927306 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:49.927208    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:49.927465 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:49.927300    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:49.927465 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:49.927097    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:49.927465 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:49.927429    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:51.069577 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:51.069539    2581 generic.go:356] "Generic (PLEG): container finished" podID="ce04ed69-ff4c-438a-9261-9f3c66fc6c09" containerID="c9c6e59c58c24349539483f47dc3098f6e6e9c9b09b6ddb7ae569aec6435ec05" exitCode=0
Jul 07 14:57:51.070056 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:51.069593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwsc4" event={"ID":"ce04ed69-ff4c-438a-9261-9f3c66fc6c09","Type":"ContainerDied","Data":"c9c6e59c58c24349539483f47dc3098f6e6e9c9b09b6ddb7ae569aec6435ec05"}
Jul 07 14:57:51.926868 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:51.926832    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:51.926868 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:51.926867    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:51.927112 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:51.926878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:51.927112 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:51.926967    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:51.927112 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:51.927089    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:51.927273 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:51.927183    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:52.452468 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:52.452437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs\") pod \"network-metrics-daemon-bjq65\" (UID: \"2c77f4a4-6fbf-4ca3-8291-a48491c2f423\") " pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:52.452930 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:52.452543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:52.452930 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:52.452620    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:52.452930 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:52.452666    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:52.452930 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:52.452703    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs podName:2c77f4a4-6fbf-4ca3-8291-a48491c2f423 nodeName:}" failed. No retries permitted until 2026-07-07 14:58:24.452681164 +0000 UTC m=+66.185293479 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs") pod "network-metrics-daemon-bjq65" (UID: "2c77f4a4-6fbf-4ca3-8291-a48491c2f423") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:52.452930 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:52.452724    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret podName:386c6191-61a2-412c-9331-152a3d94baf9 nodeName:}" failed. No retries permitted until 2026-07-07 14:58:24.452713764 +0000 UTC m=+66.185326069 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret") pod "global-pull-secret-syncer-b9d6j" (UID: "386c6191-61a2-412c-9331-152a3d94baf9") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:52.553420 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:52.553376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fppb\" (UniqueName: \"kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb\") pod \"network-check-target-qk5j7\" (UID: \"0043d8b0-5817-4b30-950b-0145e8690f3f\") " pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:52.553564 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:52.553535    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:57:52.553564 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:52.553553    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:57:52.553564 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:52.553563    2581 projected.go:196] Error preparing data for projected volume kube-api-access-7fppb for pod openshift-network-diagnostics/network-check-target-qk5j7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:52.553691 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:52.553644    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb podName:0043d8b0-5817-4b30-950b-0145e8690f3f nodeName:}" failed. No retries permitted until 2026-07-07 14:58:24.553627318 +0000 UTC m=+66.286239632 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7fppb" (UniqueName: "kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb") pod "network-check-target-qk5j7" (UID: "0043d8b0-5817-4b30-950b-0145e8690f3f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:53.926577 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:53.926537    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:53.927218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:53.926591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:53.927218 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:53.926591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:53.927218 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:53.926675    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:53.927218 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:53.926790    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:53.927218 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:53.926883    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:55.926805 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:55.926766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:55.926805 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:55.926807    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:55.927316 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:55.926807    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:55.927316 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:55.926872    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:55.927316 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:55.926992    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:55.927316 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:55.927039    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:56.082499 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:56.082457    2581 generic.go:356] "Generic (PLEG): container finished" podID="ce04ed69-ff4c-438a-9261-9f3c66fc6c09" containerID="9506c1b410b61dc89a439483058b2ae45514365eb8c6dd1bb1cfc637364b44ae" exitCode=0
Jul 07 14:57:56.082664 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:56.082524    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwsc4" event={"ID":"ce04ed69-ff4c-438a-9261-9f3c66fc6c09","Type":"ContainerDied","Data":"9506c1b410b61dc89a439483058b2ae45514365eb8c6dd1bb1cfc637364b44ae"}
Jul 07 14:57:57.088289 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:57.088258    2581 generic.go:356] "Generic (PLEG): container finished" podID="ce04ed69-ff4c-438a-9261-9f3c66fc6c09" containerID="290d29e4a3c20629e2f7d255663587fabff5f333af8bc0b0db5ff4a4c52335ff" exitCode=0
Jul 07 14:57:57.088289 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:57.088295    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwsc4" event={"ID":"ce04ed69-ff4c-438a-9261-9f3c66fc6c09","Type":"ContainerDied","Data":"290d29e4a3c20629e2f7d255663587fabff5f333af8bc0b0db5ff4a4c52335ff"}
Jul 07 14:57:57.926907 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:57.926871    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:57.926907 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:57.926904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:57.927108 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:57.926904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:57.927108 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:57.926970    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:57.927108 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:57.927062    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:57:57.927206 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:57.927129    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:58.093763 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:58.093731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwsc4" event={"ID":"ce04ed69-ff4c-438a-9261-9f3c66fc6c09","Type":"ContainerStarted","Data":"3b399bddcfe616edc5894d233aa8ab826543f1ddd096699db6b3355f368830e4"}
Jul 07 14:57:59.927479 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:59.927440    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:57:59.927479 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:59.927467    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:57:59.927994 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:57:59.927440    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:57:59.927994 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:59.927574    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:57:59.927994 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:59.927651    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:57:59.927994 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:57:59.927751    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:58:01.926673 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:01.926636    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:58:01.926673 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:01.926649    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:58:01.926673 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:01.926649    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:58:01.927149 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:01.926731    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:58:01.927149 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:01.926807    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:58:01.927149 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:01.926904    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:58:03.926621 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:03.926583    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:58:03.927040 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:03.926584    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:58:03.927040 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:03.926689    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:58:03.927040 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:03.926584    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:58:03.927040 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:03.926770    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:58:03.927040 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:03.926856    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:58:05.927042 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:05.927012    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:58:05.927444 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:05.927102    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:58:05.927444 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:05.927112    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:58:05.927444 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:05.927186    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:58:05.927444 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:05.927219    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:58:05.927444 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:05.927266    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:58:07.927178 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:07.926996    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:58:07.927616 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:07.926996    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:58:07.927616 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:07.927271    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:58:07.927616 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:07.926996    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:58:07.927616 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:07.927316    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:58:07.927616 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:07.927450    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:58:08.592309 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:08.592250    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wwsc4" podStartSLOduration=18.854865353 podStartE2EDuration="49.592234922s" totalImagesPullingTime="30.737369569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:57:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:58.136607266 +0000 UTC m=+39.869219586" watchObservedRunningTime="2026-07-07 14:58:08.592234922 +0000 UTC m=+50.324847263"
Jul 07 14:58:08.592812 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:08.592792    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qk5j7"]
Jul 07 14:58:08.592902 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:08.592886    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:58:08.593002 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:08.592966    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:58:08.598119 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:08.598090    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bjq65"]
Jul 07 14:58:08.598237 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:08.598188    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:58:08.598302 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:08.598280    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:58:08.598718 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:08.598658    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b9d6j"]
Jul 07 14:58:08.598827 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:08.598743    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:58:08.598827 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:08.598819    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:58:10.927049 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:10.927015    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:58:10.927049 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:10.927050    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:58:10.927780 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:10.927162    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjq65" podUID="2c77f4a4-6fbf-4ca3-8291-a48491c2f423"
Jul 07 14:58:10.927780 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:10.927224    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:58:10.927780 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:10.927287    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5j7" podUID="0043d8b0-5817-4b30-950b-0145e8690f3f"
Jul 07 14:58:10.927780 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:10.927212    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9d6j" podUID="386c6191-61a2-412c-9331-152a3d94baf9"
Jul 07 14:58:12.021366 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.021313    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeReady"
Jul 07 14:58:12.021964 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.021481    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 14:58:12.073533 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.073505    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gdcl2"]
Jul 07 14:58:12.096911 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.096884    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fm8rl"]
Jul 07 14:58:12.097076 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.097019    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:12.099924 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.099891    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 14:58:12.100078 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.100057    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 14:58:12.100720 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.100702    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jkfb7\""
Jul 07 14:58:12.120887 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.120855    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gdcl2"]
Jul 07 14:58:12.120887 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.120894    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fm8rl"]
Jul 07 14:58:12.121117 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.120985    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.124032 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.124003    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 14:58:12.124162 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.124055    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 14:58:12.124162 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.124097    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5pp4t\""
Jul 07 14:58:12.124258 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.124198    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 14:58:12.124694 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.124671    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 14:58:12.175680 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.175648    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lmjpm"]
Jul 07 14:58:12.195145 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.195113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5521ca14-0a45-4824-bf15-fac6fce0e2f0-metrics-tls\") pod \"dns-default-gdcl2\" (UID: \"5521ca14-0a45-4824-bf15-fac6fce0e2f0\") " pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:12.195145 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.195145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5521ca14-0a45-4824-bf15-fac6fce0e2f0-config-volume\") pod \"dns-default-gdcl2\" (UID: \"5521ca14-0a45-4824-bf15-fac6fce0e2f0\") " pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:12.195389 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.195178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5521ca14-0a45-4824-bf15-fac6fce0e2f0-tmp-dir\") pod \"dns-default-gdcl2\" (UID: \"5521ca14-0a45-4824-bf15-fac6fce0e2f0\") " pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:12.195389 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.195293    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krwxh\" (UniqueName: \"kubernetes.io/projected/5521ca14-0a45-4824-bf15-fac6fce0e2f0-kube-api-access-krwxh\") pod \"dns-default-gdcl2\" (UID: \"5521ca14-0a45-4824-bf15-fac6fce0e2f0\") " pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:12.196880 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.196850    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lmjpm"]
Jul 07 14:58:12.196984 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.196963    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lmjpm"
Jul 07 14:58:12.201857 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.201835    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 14:58:12.201987 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.201927    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 14:58:12.202269 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.202255    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 14:58:12.203395 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.203376    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kbzh7\""
Jul 07 14:58:12.296395 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.296298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzrf4\" (UniqueName: \"kubernetes.io/projected/3bf7e0cf-669e-42c4-8ee8-76764abc256a-kube-api-access-mzrf4\") pod \"ingress-canary-lmjpm\" (UID: \"3bf7e0cf-669e-42c4-8ee8-76764abc256a\") " pod="openshift-ingress-canary/ingress-canary-lmjpm"
Jul 07 14:58:12.296395 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.296369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a34eac8-0d11-49d8-bd6a-9bfec15e3144-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fm8rl\" (UID: \"2a34eac8-0d11-49d8-bd6a-9bfec15e3144\") " pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.296590 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.296435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9g4x\" (UniqueName: \"kubernetes.io/projected/2a34eac8-0d11-49d8-bd6a-9bfec15e3144-kube-api-access-j9g4x\") pod \"insights-runtime-extractor-fm8rl\" (UID: \"2a34eac8-0d11-49d8-bd6a-9bfec15e3144\") " pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.296590 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.296468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krwxh\" (UniqueName: \"kubernetes.io/projected/5521ca14-0a45-4824-bf15-fac6fce0e2f0-kube-api-access-krwxh\") pod \"dns-default-gdcl2\" (UID: \"5521ca14-0a45-4824-bf15-fac6fce0e2f0\") " pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:12.296590 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.296493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a34eac8-0d11-49d8-bd6a-9bfec15e3144-data-volume\") pod \"insights-runtime-extractor-fm8rl\" (UID: \"2a34eac8-0d11-49d8-bd6a-9bfec15e3144\") " pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.296590 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.296529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a34eac8-0d11-49d8-bd6a-9bfec15e3144-crio-socket\") pod \"insights-runtime-extractor-fm8rl\" (UID: \"2a34eac8-0d11-49d8-bd6a-9bfec15e3144\") " pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.296590 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.296576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5521ca14-0a45-4824-bf15-fac6fce0e2f0-metrics-tls\") pod \"dns-default-gdcl2\" (UID: \"5521ca14-0a45-4824-bf15-fac6fce0e2f0\") " pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:12.296778 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.296601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bf7e0cf-669e-42c4-8ee8-76764abc256a-cert\") pod \"ingress-canary-lmjpm\" (UID: \"3bf7e0cf-669e-42c4-8ee8-76764abc256a\") " pod="openshift-ingress-canary/ingress-canary-lmjpm"
Jul 07 14:58:12.296778 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.296626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5521ca14-0a45-4824-bf15-fac6fce0e2f0-config-volume\") pod \"dns-default-gdcl2\" (UID: \"5521ca14-0a45-4824-bf15-fac6fce0e2f0\") " pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:12.296778 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.296650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a34eac8-0d11-49d8-bd6a-9bfec15e3144-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fm8rl\" (UID: \"2a34eac8-0d11-49d8-bd6a-9bfec15e3144\") " pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.296778 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.296692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5521ca14-0a45-4824-bf15-fac6fce0e2f0-tmp-dir\") pod \"dns-default-gdcl2\" (UID: \"5521ca14-0a45-4824-bf15-fac6fce0e2f0\") " pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:12.297090 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.297038    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5521ca14-0a45-4824-bf15-fac6fce0e2f0-tmp-dir\") pod \"dns-default-gdcl2\" (UID: \"5521ca14-0a45-4824-bf15-fac6fce0e2f0\") " pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:12.297244 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.297228    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5521ca14-0a45-4824-bf15-fac6fce0e2f0-config-volume\") pod \"dns-default-gdcl2\" (UID: \"5521ca14-0a45-4824-bf15-fac6fce0e2f0\") " pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:12.300623 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.300604    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5521ca14-0a45-4824-bf15-fac6fce0e2f0-metrics-tls\") pod \"dns-default-gdcl2\" (UID: \"5521ca14-0a45-4824-bf15-fac6fce0e2f0\") " pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:12.305449 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.305427    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krwxh\" (UniqueName: \"kubernetes.io/projected/5521ca14-0a45-4824-bf15-fac6fce0e2f0-kube-api-access-krwxh\") pod \"dns-default-gdcl2\" (UID: \"5521ca14-0a45-4824-bf15-fac6fce0e2f0\") " pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:12.397537 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.397503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a34eac8-0d11-49d8-bd6a-9bfec15e3144-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fm8rl\" (UID: \"2a34eac8-0d11-49d8-bd6a-9bfec15e3144\") " pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.397726 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.397544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9g4x\" (UniqueName: \"kubernetes.io/projected/2a34eac8-0d11-49d8-bd6a-9bfec15e3144-kube-api-access-j9g4x\") pod \"insights-runtime-extractor-fm8rl\" (UID: \"2a34eac8-0d11-49d8-bd6a-9bfec15e3144\") " pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.397726 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.397581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a34eac8-0d11-49d8-bd6a-9bfec15e3144-data-volume\") pod \"insights-runtime-extractor-fm8rl\" (UID: \"2a34eac8-0d11-49d8-bd6a-9bfec15e3144\") " pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.397726 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.397633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a34eac8-0d11-49d8-bd6a-9bfec15e3144-crio-socket\") pod \"insights-runtime-extractor-fm8rl\" (UID: \"2a34eac8-0d11-49d8-bd6a-9bfec15e3144\") " pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.397726 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.397654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bf7e0cf-669e-42c4-8ee8-76764abc256a-cert\") pod \"ingress-canary-lmjpm\" (UID: \"3bf7e0cf-669e-42c4-8ee8-76764abc256a\") " pod="openshift-ingress-canary/ingress-canary-lmjpm"
Jul 07 14:58:12.397726 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.397670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a34eac8-0d11-49d8-bd6a-9bfec15e3144-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fm8rl\" (UID: \"2a34eac8-0d11-49d8-bd6a-9bfec15e3144\") " pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.397726 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.397704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzrf4\" (UniqueName: \"kubernetes.io/projected/3bf7e0cf-669e-42c4-8ee8-76764abc256a-kube-api-access-mzrf4\") pod \"ingress-canary-lmjpm\" (UID: \"3bf7e0cf-669e-42c4-8ee8-76764abc256a\") " pod="openshift-ingress-canary/ingress-canary-lmjpm"
Jul 07 14:58:12.398022 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.397791    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a34eac8-0d11-49d8-bd6a-9bfec15e3144-crio-socket\") pod \"insights-runtime-extractor-fm8rl\" (UID: \"2a34eac8-0d11-49d8-bd6a-9bfec15e3144\") " pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.398089 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.398067    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a34eac8-0d11-49d8-bd6a-9bfec15e3144-data-volume\") pod \"insights-runtime-extractor-fm8rl\" (UID: \"2a34eac8-0d11-49d8-bd6a-9bfec15e3144\") " pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.398281 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.398263    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a34eac8-0d11-49d8-bd6a-9bfec15e3144-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fm8rl\" (UID: \"2a34eac8-0d11-49d8-bd6a-9bfec15e3144\") " pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.399997 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.399975    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a34eac8-0d11-49d8-bd6a-9bfec15e3144-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fm8rl\" (UID: \"2a34eac8-0d11-49d8-bd6a-9bfec15e3144\") " pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.400615 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.400596    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bf7e0cf-669e-42c4-8ee8-76764abc256a-cert\") pod \"ingress-canary-lmjpm\" (UID: \"3bf7e0cf-669e-42c4-8ee8-76764abc256a\") " pod="openshift-ingress-canary/ingress-canary-lmjpm"
Jul 07 14:58:12.407521 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.407500    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:12.413647 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.413623    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzrf4\" (UniqueName: \"kubernetes.io/projected/3bf7e0cf-669e-42c4-8ee8-76764abc256a-kube-api-access-mzrf4\") pod \"ingress-canary-lmjpm\" (UID: \"3bf7e0cf-669e-42c4-8ee8-76764abc256a\") " pod="openshift-ingress-canary/ingress-canary-lmjpm"
Jul 07 14:58:12.414914 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.414897    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9g4x\" (UniqueName: \"kubernetes.io/projected/2a34eac8-0d11-49d8-bd6a-9bfec15e3144-kube-api-access-j9g4x\") pod \"insights-runtime-extractor-fm8rl\" (UID: \"2a34eac8-0d11-49d8-bd6a-9bfec15e3144\") " pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.429294 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.429264    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fm8rl"
Jul 07 14:58:12.505271 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.505103    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lmjpm"
Jul 07 14:58:12.665557 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.665527    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gdcl2"]
Jul 07 14:58:12.669278 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:58:12.669249    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5521ca14_0a45_4824_bf15_fac6fce0e2f0.slice/crio-20872e27ba6ab6fcfaf65723ba222f6b4d12f902ce32bc9c943e33ccdb60d9bc WatchSource:0}: Error finding container 20872e27ba6ab6fcfaf65723ba222f6b4d12f902ce32bc9c943e33ccdb60d9bc: Status 404 returned error can't find the container with id 20872e27ba6ab6fcfaf65723ba222f6b4d12f902ce32bc9c943e33ccdb60d9bc
Jul 07 14:58:12.679021 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.676728    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fm8rl"]
Jul 07 14:58:12.687316 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.687282    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lmjpm"]
Jul 07 14:58:12.690733 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:58:12.690706    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf7e0cf_669e_42c4_8ee8_76764abc256a.slice/crio-ecde59b23e24c28a5b85325500ca3230e2b8fc4ba5cc3dc1758a658947a88436 WatchSource:0}: Error finding container ecde59b23e24c28a5b85325500ca3230e2b8fc4ba5cc3dc1758a658947a88436: Status 404 returned error can't find the container with id ecde59b23e24c28a5b85325500ca3230e2b8fc4ba5cc3dc1758a658947a88436
Jul 07 14:58:12.927477 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.927438    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:58:12.927666 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.927438    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:58:12.927666 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.927438    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:58:12.930798 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.930776    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 14:58:12.932718 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.932696    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kb9sm\""
Jul 07 14:58:12.932852 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.932815    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 14:58:12.932923 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.932897    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 14:58:12.932977 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.932936    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4xq58\""
Jul 07 14:58:12.933127 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:12.932990    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 14:58:13.129602 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:13.129560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gdcl2" event={"ID":"5521ca14-0a45-4824-bf15-fac6fce0e2f0","Type":"ContainerStarted","Data":"20872e27ba6ab6fcfaf65723ba222f6b4d12f902ce32bc9c943e33ccdb60d9bc"}
Jul 07 14:58:13.130830 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:13.130802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lmjpm" event={"ID":"3bf7e0cf-669e-42c4-8ee8-76764abc256a","Type":"ContainerStarted","Data":"ecde59b23e24c28a5b85325500ca3230e2b8fc4ba5cc3dc1758a658947a88436"}
Jul 07 14:58:13.131949 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:13.131927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fm8rl" event={"ID":"2a34eac8-0d11-49d8-bd6a-9bfec15e3144","Type":"ContainerStarted","Data":"b8a1b629ce67f55e62359991e72fe426bf47a79f4d90caab72de4be2eefd7353"}
Jul 07 14:58:13.469242 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:13.469206    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bzfwg"]
Jul 07 14:58:13.471634 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:13.471608    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bzfwg"
Jul 07 14:58:13.475711 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:13.475684    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 07 14:58:13.476002 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:13.475983    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5jtf9\""
Jul 07 14:58:13.493966 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:13.493929    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bzfwg"]
Jul 07 14:58:13.607844 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:13.607749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0277897e-a284-4a9c-84a1-3e05e767f3e6-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-bzfwg\" (UID: \"0277897e-a284-4a9c-84a1-3e05e767f3e6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bzfwg"
Jul 07 14:58:13.709013 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:13.708491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0277897e-a284-4a9c-84a1-3e05e767f3e6-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-bzfwg\" (UID: \"0277897e-a284-4a9c-84a1-3e05e767f3e6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bzfwg"
Jul 07 14:58:13.709013 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:13.708692    2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 07 14:58:13.709013 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:13.708771    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0277897e-a284-4a9c-84a1-3e05e767f3e6-tls-certificates podName:0277897e-a284-4a9c-84a1-3e05e767f3e6 nodeName:}" failed. No retries permitted until 2026-07-07 14:58:14.2087505 +0000 UTC m=+55.941362799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/0277897e-a284-4a9c-84a1-3e05e767f3e6-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-bzfwg" (UID: "0277897e-a284-4a9c-84a1-3e05e767f3e6") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 07 14:58:14.213589 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:14.213553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0277897e-a284-4a9c-84a1-3e05e767f3e6-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-bzfwg\" (UID: \"0277897e-a284-4a9c-84a1-3e05e767f3e6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bzfwg"
Jul 07 14:58:14.221023 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:14.220990    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0277897e-a284-4a9c-84a1-3e05e767f3e6-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-bzfwg\" (UID: \"0277897e-a284-4a9c-84a1-3e05e767f3e6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bzfwg"
Jul 07 14:58:14.394552 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:14.394516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bzfwg"
Jul 07 14:58:14.574056 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:14.574024    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bzfwg"]
Jul 07 14:58:16.271497 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:58:16.271460    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0277897e_a284_4a9c_84a1_3e05e767f3e6.slice/crio-3b5a35c5a64ab6e39faf2f35af47db4b68e59d9d623085a9263eab6958b057ef WatchSource:0}: Error finding container 3b5a35c5a64ab6e39faf2f35af47db4b68e59d9d623085a9263eab6958b057ef: Status 404 returned error can't find the container with id 3b5a35c5a64ab6e39faf2f35af47db4b68e59d9d623085a9263eab6958b057ef
Jul 07 14:58:17.146258 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:17.146219    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bzfwg" event={"ID":"0277897e-a284-4a9c-84a1-3e05e767f3e6","Type":"ContainerStarted","Data":"3b5a35c5a64ab6e39faf2f35af47db4b68e59d9d623085a9263eab6958b057ef"}
Jul 07 14:58:19.075685 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.075654    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q5z8m"
Jul 07 14:58:19.152897 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.152865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gdcl2" event={"ID":"5521ca14-0a45-4824-bf15-fac6fce0e2f0","Type":"ContainerStarted","Data":"c13be9be7c2ad849892e14f9000cc2416eff378d6189d2ca56ec98b5b99a3999"}
Jul 07 14:58:19.153021 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.152931    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:19.153021 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.152957    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gdcl2" event={"ID":"5521ca14-0a45-4824-bf15-fac6fce0e2f0","Type":"ContainerStarted","Data":"1e0fdaa83146bb0ebbba5fc5213570bdfbca925343b513364e9127df1bc43236"}
Jul 07 14:58:19.154316 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.154289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lmjpm" event={"ID":"3bf7e0cf-669e-42c4-8ee8-76764abc256a","Type":"ContainerStarted","Data":"5bf77a1e1ba4b1844ce00cdda8176d06c8d49a06b94740a5a4350ba0a5a85898"}
Jul 07 14:58:19.155745 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.155717    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bzfwg" event={"ID":"0277897e-a284-4a9c-84a1-3e05e767f3e6","Type":"ContainerStarted","Data":"19c62aaf7cdb44600d77c891dd263287e3d716880f812168846beaccbf7cd190"}
Jul 07 14:58:19.155859 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.155847    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bzfwg"
Jul 07 14:58:19.157168 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.157150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fm8rl" event={"ID":"2a34eac8-0d11-49d8-bd6a-9bfec15e3144","Type":"ContainerStarted","Data":"27abdd50d0c0c4c057556949a35f83b464c3903620572647b9d2c67b141d7861"}
Jul 07 14:58:19.161020 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.160998    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bzfwg"
Jul 07 14:58:19.172981 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.172942    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gdcl2" podStartSLOduration=1.924890875 podStartE2EDuration="7.172929185s" totalImagesPullingTime="5.24803831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:19.172273758 +0000 UTC m=+60.904886078" watchObservedRunningTime="2026-07-07 14:58:19.172929185 +0000 UTC m=+60.905541505"
Jul 07 14:58:19.190201 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.190154    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lmjpm" podStartSLOduration=1.9634594459999999 podStartE2EDuration="7.190139749s" totalImagesPullingTime="5.226680303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:19.189323223 +0000 UTC m=+60.921935545" watchObservedRunningTime="2026-07-07 14:58:19.190139749 +0000 UTC m=+60.922752068"
Jul 07 14:58:19.211535 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.211480    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bzfwg" podStartSLOduration=4.289449259 podStartE2EDuration="6.211459405s" totalImagesPullingTime="1.922010146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:19.210994685 +0000 UTC m=+60.943607004" watchObservedRunningTime="2026-07-07 14:58:19.211459405 +0000 UTC m=+60.944071722"
Jul 07 14:58:19.542478 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.542448    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"]
Jul 07 14:58:19.556631 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.556606    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"
Jul 07 14:58:19.557979 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.557951    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"]
Jul 07 14:58:19.560436 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.560409    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 07 14:58:19.560573 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.560443    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 07 14:58:19.560573 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.560484    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 14:58:19.560862 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.560843    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 14:58:19.561044 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.560861    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4lqsb\""
Jul 07 14:58:19.562138 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.562119    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 14:58:19.656233 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.656196    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-jdh4f\" (UID: \"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"
Jul 07 14:58:19.656233 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.656237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqdx6\" (UniqueName: \"kubernetes.io/projected/754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6-kube-api-access-jqdx6\") pod \"prometheus-operator-567f7b8b7c-jdh4f\" (UID: \"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"
Jul 07 14:58:19.656500 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.656269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-jdh4f\" (UID: \"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"
Jul 07 14:58:19.656500 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.656338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-jdh4f\" (UID: \"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"
Jul 07 14:58:19.757733 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.757694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-jdh4f\" (UID: \"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"
Jul 07 14:58:19.757923 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.757763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-jdh4f\" (UID: \"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"
Jul 07 14:58:19.757923 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.757796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqdx6\" (UniqueName: \"kubernetes.io/projected/754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6-kube-api-access-jqdx6\") pod \"prometheus-operator-567f7b8b7c-jdh4f\" (UID: \"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"
Jul 07 14:58:19.757923 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.757826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-jdh4f\" (UID: \"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"
Jul 07 14:58:19.758098 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:19.757954    2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 07 14:58:19.758098 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:19.758050    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6-prometheus-operator-tls podName:754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6 nodeName:}" failed. No retries permitted until 2026-07-07 14:58:20.258029937 +0000 UTC m=+61.990642253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-jdh4f" (UID: "754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6") : secret "prometheus-operator-tls" not found
Jul 07 14:58:19.758533 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.758501    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-jdh4f\" (UID: \"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"
Jul 07 14:58:19.760407 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.760388    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-jdh4f\" (UID: \"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"
Jul 07 14:58:19.767715 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:19.767694    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqdx6\" (UniqueName: \"kubernetes.io/projected/754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6-kube-api-access-jqdx6\") pod \"prometheus-operator-567f7b8b7c-jdh4f\" (UID: \"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"
Jul 07 14:58:20.162124 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:20.162084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fm8rl" event={"ID":"2a34eac8-0d11-49d8-bd6a-9bfec15e3144","Type":"ContainerStarted","Data":"285cd770d60207c08ea08be38f33d8d954e81070269d758e2e224d9e7b015bec"}
Jul 07 14:58:20.261677 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:20.261619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-jdh4f\" (UID: \"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"
Jul 07 14:58:20.264568 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:20.264543    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-jdh4f\" (UID: \"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"
Jul 07 14:58:20.466543 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:20.466513    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"
Jul 07 14:58:20.604462 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:20.604416    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f"]
Jul 07 14:58:20.609805 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:58:20.609776    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754b8f0b_4a44_4d63_aa46_2d1cc4d6a6f6.slice/crio-f657acb3aea41f00589d3cc0d750ee766fc9fd0083e3f4edeb064dced6b90f54 WatchSource:0}: Error finding container f657acb3aea41f00589d3cc0d750ee766fc9fd0083e3f4edeb064dced6b90f54: Status 404 returned error can't find the container with id f657acb3aea41f00589d3cc0d750ee766fc9fd0083e3f4edeb064dced6b90f54
Jul 07 14:58:21.169274 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:21.169198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f" event={"ID":"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6","Type":"ContainerStarted","Data":"f657acb3aea41f00589d3cc0d750ee766fc9fd0083e3f4edeb064dced6b90f54"}
Jul 07 14:58:22.174512 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:22.174477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fm8rl" event={"ID":"2a34eac8-0d11-49d8-bd6a-9bfec15e3144","Type":"ContainerStarted","Data":"68674adb33dadb18535bd95bec453b862eb5033b437439fd7f4aca11606504d9"}
Jul 07 14:58:22.197439 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:22.197394    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fm8rl" podStartSLOduration=0.972615596 podStartE2EDuration="10.197379677s" totalImagesPullingTime="9.224764081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:58:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:22.195732763 +0000 UTC m=+63.928345082" watchObservedRunningTime="2026-07-07 14:58:22.197379677 +0000 UTC m=+63.929991997"
Jul 07 14:58:23.179759 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:23.179727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f" event={"ID":"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6","Type":"ContainerStarted","Data":"c3f90662729b50019ac7c650739a966b2e0fdf6b7005de9fef0c2956169ed512"}
Jul 07 14:58:23.180134 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:23.179778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f" event={"ID":"754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6","Type":"ContainerStarted","Data":"1d352732d88cfe926915b15cdca72b5a6958757a261f38764c2ea8ce9068b0e9"}
Jul 07 14:58:23.201667 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:23.201624    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jdh4f" podStartSLOduration=1.936302325 podStartE2EDuration="4.201608306s" totalImagesPullingTime="2.265305981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:23.199921511 +0000 UTC m=+64.932533830" watchObservedRunningTime="2026-07-07 14:58:23.201608306 +0000 UTC m=+64.934220627"
Jul 07 14:58:24.495106 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.495058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs\") pod \"network-metrics-daemon-bjq65\" (UID: \"2c77f4a4-6fbf-4ca3-8291-a48491c2f423\") " pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:58:24.495541 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.495133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:58:24.498055 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.498037    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 14:58:24.498107 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.498067    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 14:58:24.507393 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.507370    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/386c6191-61a2-412c-9331-152a3d94baf9-original-pull-secret\") pod \"global-pull-secret-syncer-b9d6j\" (UID: \"386c6191-61a2-412c-9331-152a3d94baf9\") " pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:58:24.507522 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.507443    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c77f4a4-6fbf-4ca3-8291-a48491c2f423-metrics-certs\") pod \"network-metrics-daemon-bjq65\" (UID: \"2c77f4a4-6fbf-4ca3-8291-a48491c2f423\") " pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:58:24.595716 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.595676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fppb\" (UniqueName: \"kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb\") pod \"network-check-target-qk5j7\" (UID: \"0043d8b0-5817-4b30-950b-0145e8690f3f\") " pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:58:24.599630 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.599505    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 14:58:24.609106 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.609077    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 14:58:24.620362 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.620324    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fppb\" (UniqueName: \"kubernetes.io/projected/0043d8b0-5817-4b30-950b-0145e8690f3f-kube-api-access-7fppb\") pod \"network-check-target-qk5j7\" (UID: \"0043d8b0-5817-4b30-950b-0145e8690f3f\") " pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:58:24.642846 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.642822    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kb9sm\""
Jul 07 14:58:24.643895 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.643878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9d6j"
Jul 07 14:58:24.649112 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.649087    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:58:24.651591 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.651572    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4xq58\""
Jul 07 14:58:24.658172 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.658066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjq65"
Jul 07 14:58:24.785300 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.785227    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b9d6j"]
Jul 07 14:58:24.790131 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:58:24.790098    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386c6191_61a2_412c_9331_152a3d94baf9.slice/crio-990ebe495ccd709762d083b74a6884f34f6f93458cf9eba734e04bc27da87b31 WatchSource:0}: Error finding container 990ebe495ccd709762d083b74a6884f34f6f93458cf9eba734e04bc27da87b31: Status 404 returned error can't find the container with id 990ebe495ccd709762d083b74a6884f34f6f93458cf9eba734e04bc27da87b31
Jul 07 14:58:24.804148 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.804123    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qk5j7"]
Jul 07 14:58:24.806684 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:58:24.806658    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0043d8b0_5817_4b30_950b_0145e8690f3f.slice/crio-bcc53e3046bba60add0eb11e20a0e28f782181d09cfdfbc8de7d9f0b760319d9 WatchSource:0}: Error finding container bcc53e3046bba60add0eb11e20a0e28f782181d09cfdfbc8de7d9f0b760319d9: Status 404 returned error can't find the container with id bcc53e3046bba60add0eb11e20a0e28f782181d09cfdfbc8de7d9f0b760319d9
Jul 07 14:58:24.823472 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.823450    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bjq65"]
Jul 07 14:58:24.826703 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:58:24.826679    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c77f4a4_6fbf_4ca3_8291_a48491c2f423.slice/crio-3bf0141a5f31c3f80eee8f0d9c19c80112e918099bb4532c8fced3673c92b12c WatchSource:0}: Error finding container 3bf0141a5f31c3f80eee8f0d9c19c80112e918099bb4532c8fced3673c92b12c: Status 404 returned error can't find the container with id 3bf0141a5f31c3f80eee8f0d9c19c80112e918099bb4532c8fced3673c92b12c
Jul 07 14:58:24.917769 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.917740    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"]
Jul 07 14:58:24.920592 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.920576    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:24.924145 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.924081    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gpw7r\""
Jul 07 14:58:24.924333 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.924317    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 07 14:58:24.924649 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.924633    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 07 14:58:24.938301 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.938275    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"]
Jul 07 14:58:24.939083 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.939055    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tq54m"]
Jul 07 14:58:24.941114 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.941095    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"]
Jul 07 14:58:24.941274 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.941257    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:24.943402 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.943383    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:24.944203 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.944185    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 14:58:24.944298 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.944237    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 14:58:24.944534 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.944517    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 14:58:24.944619 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.944603    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4wp8w\""
Jul 07 14:58:24.946332 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.946310    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 07 14:58:24.946436 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.946311    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-r4k48\""
Jul 07 14:58:24.946682 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.946650    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 07 14:58:24.946801 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.946741    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 07 14:58:24.959892 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.959869    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"]
Jul 07 14:58:24.999068 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2e6d5ccf-86b8-49d2-9a73-835584742a65-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:24.999068 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999071    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6f7c\" (UniqueName: \"kubernetes.io/projected/2e6d5ccf-86b8-49d2-9a73-835584742a65-kube-api-access-p6f7c\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:24.999266 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/334b7bba-a8fa-4589-98db-4f59626c7da6-metrics-client-ca\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:24.999266 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e6d5ccf-86b8-49d2-9a73-835584742a65-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:24.999266 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5cgw\" (UniqueName: \"kubernetes.io/projected/334b7bba-a8fa-4589-98db-4f59626c7da6-kube-api-access-x5cgw\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:24.999266 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/334b7bba-a8fa-4589-98db-4f59626c7da6-root\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:24.999266 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/334b7bba-a8fa-4589-98db-4f59626c7da6-sys\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:24.999266 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-textfile\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:24.999266 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/faad4c48-420e-4d7d-9673-1d63790e2c33-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-xbxlx\" (UID: \"faad4c48-420e-4d7d-9673-1d63790e2c33\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:24.999540 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbg7r\" (UniqueName: \"kubernetes.io/projected/faad4c48-420e-4d7d-9673-1d63790e2c33-kube-api-access-gbg7r\") pod \"openshift-state-metrics-54cf96dfcf-xbxlx\" (UID: \"faad4c48-420e-4d7d-9673-1d63790e2c33\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:24.999540 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e6d5ccf-86b8-49d2-9a73-835584742a65-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:24.999540 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-wtmp\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:24.999540 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-accelerators-collector-config\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:24.999540 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e6d5ccf-86b8-49d2-9a73-835584742a65-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:24.999540 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faad4c48-420e-4d7d-9673-1d63790e2c33-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-xbxlx\" (UID: \"faad4c48-420e-4d7d-9673-1d63790e2c33\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:24.999540 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-tls\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:24.999540 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2e6d5ccf-86b8-49d2-9a73-835584742a65-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:24.999540 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faad4c48-420e-4d7d-9673-1d63790e2c33-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-xbxlx\" (UID: \"faad4c48-420e-4d7d-9673-1d63790e2c33\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:24.999540 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:24.999501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.100205 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5cgw\" (UniqueName: \"kubernetes.io/projected/334b7bba-a8fa-4589-98db-4f59626c7da6-kube-api-access-x5cgw\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.100205 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/334b7bba-a8fa-4589-98db-4f59626c7da6-root\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.100205 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/334b7bba-a8fa-4589-98db-4f59626c7da6-sys\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.100205 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-textfile\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.100205 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/faad4c48-420e-4d7d-9673-1d63790e2c33-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-xbxlx\" (UID: \"faad4c48-420e-4d7d-9673-1d63790e2c33\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:25.100664 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbg7r\" (UniqueName: \"kubernetes.io/projected/faad4c48-420e-4d7d-9673-1d63790e2c33-kube-api-access-gbg7r\") pod \"openshift-state-metrics-54cf96dfcf-xbxlx\" (UID: \"faad4c48-420e-4d7d-9673-1d63790e2c33\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:25.100664 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100227    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/334b7bba-a8fa-4589-98db-4f59626c7da6-sys\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.100664 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e6d5ccf-86b8-49d2-9a73-835584742a65-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:25.100664 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-wtmp\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.100664 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:25.100311    2581 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 07 14:58:25.100664 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100227    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/334b7bba-a8fa-4589-98db-4f59626c7da6-root\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.100664 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-accelerators-collector-config\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.100664 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:25.100392    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faad4c48-420e-4d7d-9673-1d63790e2c33-openshift-state-metrics-tls podName:faad4c48-420e-4d7d-9673-1d63790e2c33 nodeName:}" failed. No retries permitted until 2026-07-07 14:58:25.600372029 +0000 UTC m=+67.332984347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/faad4c48-420e-4d7d-9673-1d63790e2c33-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-xbxlx" (UID: "faad4c48-420e-4d7d-9673-1d63790e2c33") : secret "openshift-state-metrics-tls" not found
Jul 07 14:58:25.100664 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e6d5ccf-86b8-49d2-9a73-835584742a65-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:25.100664 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faad4c48-420e-4d7d-9673-1d63790e2c33-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-xbxlx\" (UID: \"faad4c48-420e-4d7d-9673-1d63790e2c33\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:25.100664 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-tls\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.100664 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100514    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2e6d5ccf-86b8-49d2-9a73-835584742a65-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:25.100664 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faad4c48-420e-4d7d-9673-1d63790e2c33-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-xbxlx\" (UID: \"faad4c48-420e-4d7d-9673-1d63790e2c33\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:25.100664 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100543    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-wtmp\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.100664 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.101433 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2e6d5ccf-86b8-49d2-9a73-835584742a65-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:25.101433 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100636    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-textfile\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.101433 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6f7c\" (UniqueName: \"kubernetes.io/projected/2e6d5ccf-86b8-49d2-9a73-835584742a65-kube-api-access-p6f7c\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:25.101433 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/334b7bba-a8fa-4589-98db-4f59626c7da6-metrics-client-ca\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.101433 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:25.100668    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 14:58:25.101433 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e6d5ccf-86b8-49d2-9a73-835584742a65-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:25.101433 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:25.100779    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-tls podName:334b7bba-a8fa-4589-98db-4f59626c7da6 nodeName:}" failed. No retries permitted until 2026-07-07 14:58:25.600740047 +0000 UTC m=+67.333352359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-tls") pod "node-exporter-tq54m" (UID: "334b7bba-a8fa-4589-98db-4f59626c7da6") : secret "node-exporter-tls" not found
Jul 07 14:58:25.101433 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.101398    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2e6d5ccf-86b8-49d2-9a73-835584742a65-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:25.101809 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.101657    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faad4c48-420e-4d7d-9673-1d63790e2c33-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-xbxlx\" (UID: \"faad4c48-420e-4d7d-9673-1d63790e2c33\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:25.101809 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.100992    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-accelerators-collector-config\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.101908 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.101826    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2e6d5ccf-86b8-49d2-9a73-835584742a65-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:25.101908 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.101834    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/334b7bba-a8fa-4589-98db-4f59626c7da6-metrics-client-ca\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.102185 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.102151    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e6d5ccf-86b8-49d2-9a73-835584742a65-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:25.103233 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.103206    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e6d5ccf-86b8-49d2-9a73-835584742a65-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:25.103233 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.103232    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e6d5ccf-86b8-49d2-9a73-835584742a65-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:25.103726 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.103710    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.103970 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.103947    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faad4c48-420e-4d7d-9673-1d63790e2c33-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-xbxlx\" (UID: \"faad4c48-420e-4d7d-9673-1d63790e2c33\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:25.112766 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.112740    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5cgw\" (UniqueName: \"kubernetes.io/projected/334b7bba-a8fa-4589-98db-4f59626c7da6-kube-api-access-x5cgw\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.113390 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.113370    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6f7c\" (UniqueName: \"kubernetes.io/projected/2e6d5ccf-86b8-49d2-9a73-835584742a65-kube-api-access-p6f7c\") pod \"kube-state-metrics-7c9dbb8b7d-5lx8n\" (UID: \"2e6d5ccf-86b8-49d2-9a73-835584742a65\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:25.117080 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.117062    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbg7r\" (UniqueName: \"kubernetes.io/projected/faad4c48-420e-4d7d-9673-1d63790e2c33-kube-api-access-gbg7r\") pod \"openshift-state-metrics-54cf96dfcf-xbxlx\" (UID: \"faad4c48-420e-4d7d-9673-1d63790e2c33\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:25.187225 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.187192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qk5j7" event={"ID":"0043d8b0-5817-4b30-950b-0145e8690f3f","Type":"ContainerStarted","Data":"bcc53e3046bba60add0eb11e20a0e28f782181d09cfdfbc8de7d9f0b760319d9"}
Jul 07 14:58:25.188439 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.188403    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bjq65" event={"ID":"2c77f4a4-6fbf-4ca3-8291-a48491c2f423","Type":"ContainerStarted","Data":"3bf0141a5f31c3f80eee8f0d9c19c80112e918099bb4532c8fced3673c92b12c"}
Jul 07 14:58:25.189397 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.189377    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b9d6j" event={"ID":"386c6191-61a2-412c-9331-152a3d94baf9","Type":"ContainerStarted","Data":"990ebe495ccd709762d083b74a6884f34f6f93458cf9eba734e04bc27da87b31"}
Jul 07 14:58:25.256115 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.256084    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"
Jul 07 14:58:25.411638 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.411399    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n"]
Jul 07 14:58:25.414135 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:58:25.414105    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6d5ccf_86b8_49d2_9a73_835584742a65.slice/crio-e3d2039c8840da928060077498c72a1d28981b0cf22eb93c8a11950441eba109 WatchSource:0}: Error finding container e3d2039c8840da928060077498c72a1d28981b0cf22eb93c8a11950441eba109: Status 404 returned error can't find the container with id e3d2039c8840da928060077498c72a1d28981b0cf22eb93c8a11950441eba109
Jul 07 14:58:25.604852 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.604788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/faad4c48-420e-4d7d-9673-1d63790e2c33-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-xbxlx\" (UID: \"faad4c48-420e-4d7d-9673-1d63790e2c33\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:25.605328 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:25.604944    2581 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 07 14:58:25.605328 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:25.605028    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faad4c48-420e-4d7d-9673-1d63790e2c33-openshift-state-metrics-tls podName:faad4c48-420e-4d7d-9673-1d63790e2c33 nodeName:}" failed. No retries permitted until 2026-07-07 14:58:26.605005803 +0000 UTC m=+68.337618103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/faad4c48-420e-4d7d-9673-1d63790e2c33-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-xbxlx" (UID: "faad4c48-420e-4d7d-9673-1d63790e2c33") : secret "openshift-state-metrics-tls" not found
Jul 07 14:58:25.605856 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:25.605571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-tls\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:25.605856 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:25.605767    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 14:58:25.605856 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:25.605833    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-tls podName:334b7bba-a8fa-4589-98db-4f59626c7da6 nodeName:}" failed. No retries permitted until 2026-07-07 14:58:26.605817257 +0000 UTC m=+68.338429576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-tls") pod "node-exporter-tq54m" (UID: "334b7bba-a8fa-4589-98db-4f59626c7da6") : secret "node-exporter-tls" not found
Jul 07 14:58:26.000783 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.000741    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 14:58:26.015481 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.015450    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.018946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.018914    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 07 14:58:26.018946 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.018932    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 07 14:58:26.019146 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.019079    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 07 14:58:26.019223 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.019152    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hrt66\""
Jul 07 14:58:26.019759 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.019526    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 07 14:58:26.019759 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.019571    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 07 14:58:26.019759 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.019580    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 07 14:58:26.019759 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.019591    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 07 14:58:26.019759 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.019533    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 07 14:58:26.019759 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.019532    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 07 14:58:26.023822 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.023797    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 14:58:26.110751 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.110683    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.110751 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.110738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.110995 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.110766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkc7z\" (UniqueName: \"kubernetes.io/projected/329e599d-b769-4855-940e-2dbb2f95bf2b-kube-api-access-jkc7z\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.110995 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.110797    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/329e599d-b769-4855-940e-2dbb2f95bf2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.110995 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.110858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/329e599d-b769-4855-940e-2dbb2f95bf2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.110995 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.110903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/329e599d-b769-4855-940e-2dbb2f95bf2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.110995 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.110927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.110995 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.110958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/329e599d-b769-4855-940e-2dbb2f95bf2b-config-out\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.110995 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.110992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.111386 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.111017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/329e599d-b769-4855-940e-2dbb2f95bf2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.111386 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.111083    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-web-config\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.111386 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.111117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.111386 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.111148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.194834 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.194736    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n" event={"ID":"2e6d5ccf-86b8-49d2-9a73-835584742a65","Type":"ContainerStarted","Data":"e3d2039c8840da928060077498c72a1d28981b0cf22eb93c8a11950441eba109"}
Jul 07 14:58:26.212387 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.212259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-web-config\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.212387 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.212315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.213031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.212508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.213031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.212607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.213031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.212637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.213031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.212663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkc7z\" (UniqueName: \"kubernetes.io/projected/329e599d-b769-4855-940e-2dbb2f95bf2b-kube-api-access-jkc7z\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.213031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.212693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/329e599d-b769-4855-940e-2dbb2f95bf2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.213031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.212734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/329e599d-b769-4855-940e-2dbb2f95bf2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.213031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.212777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/329e599d-b769-4855-940e-2dbb2f95bf2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.213031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.212801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.213031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.212826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/329e599d-b769-4855-940e-2dbb2f95bf2b-config-out\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.213031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.212854    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.213031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.212879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/329e599d-b769-4855-940e-2dbb2f95bf2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.214146 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.213337    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/329e599d-b769-4855-940e-2dbb2f95bf2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.214146 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.213662    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/329e599d-b769-4855-940e-2dbb2f95bf2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.214294 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.214268    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/329e599d-b769-4855-940e-2dbb2f95bf2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.217447 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.217422    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.217854 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.217830    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.218647 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.218602    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.219203 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.219177    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/329e599d-b769-4855-940e-2dbb2f95bf2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.219522 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.219460    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.219522 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.219472    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-web-config\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.219685 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.219618    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.219811 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.219789    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/329e599d-b769-4855-940e-2dbb2f95bf2b-config-out\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.220514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.220484    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/329e599d-b769-4855-940e-2dbb2f95bf2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.230390 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.230257    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkc7z\" (UniqueName: \"kubernetes.io/projected/329e599d-b769-4855-940e-2dbb2f95bf2b-kube-api-access-jkc7z\") pod \"alertmanager-main-0\" (UID: \"329e599d-b769-4855-940e-2dbb2f95bf2b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.330901 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.330675    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 14:58:26.535037 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.533770    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 14:58:26.541738 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:58:26.541700    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329e599d_b769_4855_940e_2dbb2f95bf2b.slice/crio-5324aaa3e5d6dbb8c87c2173348795103a570f942e88c307231429ba9b5e6d4c WatchSource:0}: Error finding container 5324aaa3e5d6dbb8c87c2173348795103a570f942e88c307231429ba9b5e6d4c: Status 404 returned error can't find the container with id 5324aaa3e5d6dbb8c87c2173348795103a570f942e88c307231429ba9b5e6d4c
Jul 07 14:58:26.617770 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.617293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/faad4c48-420e-4d7d-9673-1d63790e2c33-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-xbxlx\" (UID: \"faad4c48-420e-4d7d-9673-1d63790e2c33\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:26.617770 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.617383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-tls\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:26.622470 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.622406    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/faad4c48-420e-4d7d-9673-1d63790e2c33-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-xbxlx\" (UID: \"faad4c48-420e-4d7d-9673-1d63790e2c33\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:26.627566 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.627510    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/334b7bba-a8fa-4589-98db-4f59626c7da6-node-exporter-tls\") pod \"node-exporter-tq54m\" (UID: \"334b7bba-a8fa-4589-98db-4f59626c7da6\") " pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:26.729399 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.729265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"
Jul 07 14:58:26.752204 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.751995    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tq54m"
Jul 07 14:58:26.784039 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:58:26.784003    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334b7bba_a8fa_4589_98db_4f59626c7da6.slice/crio-cce7485d9bfe9804bfd2aabf646d4d13d61da4ac1a2b8df2818e87571aac216e WatchSource:0}: Error finding container cce7485d9bfe9804bfd2aabf646d4d13d61da4ac1a2b8df2818e87571aac216e: Status 404 returned error can't find the container with id cce7485d9bfe9804bfd2aabf646d4d13d61da4ac1a2b8df2818e87571aac216e
Jul 07 14:58:26.912413 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.910898    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"]
Jul 07 14:58:26.923197 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:58:26.923156    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaad4c48_420e_4d7d_9673_1d63790e2c33.slice/crio-ff1b094257ad1aae82bc1d1a6f61e0b89bd9fc80a6db439625e408cf363d537b WatchSource:0}: Error finding container ff1b094257ad1aae82bc1d1a6f61e0b89bd9fc80a6db439625e408cf363d537b: Status 404 returned error can't find the container with id ff1b094257ad1aae82bc1d1a6f61e0b89bd9fc80a6db439625e408cf363d537b
Jul 07 14:58:26.929459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.929427    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:26.933843 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.933817    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 07 14:58:26.934072 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.934054    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9oh6vss0ml4d2\""
Jul 07 14:58:26.934293 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.934273    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-klvdl\""
Jul 07 14:58:26.934379 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.934368    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 07 14:58:26.934497 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.934480    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 07 14:58:26.935055 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.934585    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 07 14:58:26.935055 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.934866    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 07 14:58:26.937131 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.936403    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx"]
Jul 07 14:58:26.937131 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:26.936459    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"]
Jul 07 14:58:27.024104 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.024075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.024247 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.024122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.024247 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.024146    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-grpc-tls\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.024247 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.024179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.024455 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.024286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.024455 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.024416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-thanos-querier-tls\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.024455 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.024448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-metrics-client-ca\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.024707 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.024477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7jlw\" (UniqueName: \"kubernetes.io/projected/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-kube-api-access-h7jlw\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.125385 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.125288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-thanos-querier-tls\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.125385 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.125336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-metrics-client-ca\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.125385 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.125378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7jlw\" (UniqueName: \"kubernetes.io/projected/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-kube-api-access-h7jlw\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.125714 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.125414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.125714 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.125452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.125714 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.125482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-grpc-tls\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.125714 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.125511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.125714 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.125560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.127137 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.127110    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-metrics-client-ca\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.128886 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.128852    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.129197 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.129164    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.129447 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.129428    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-thanos-querier-tls\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.129790 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.129749    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.130911 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.130883    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.132045 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.132021    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-secret-grpc-tls\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.136539 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.136499    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7jlw\" (UniqueName: \"kubernetes.io/projected/504ebd6e-ecc5-4a6a-9afc-59648f933fb4-kube-api-access-h7jlw\") pod \"thanos-querier-5f6f9bc8bb-lspfk\" (UID: \"504ebd6e-ecc5-4a6a-9afc-59648f933fb4\") " pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:27.201813 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.201757    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"329e599d-b769-4855-940e-2dbb2f95bf2b","Type":"ContainerStarted","Data":"5324aaa3e5d6dbb8c87c2173348795103a570f942e88c307231429ba9b5e6d4c"}
Jul 07 14:58:27.204325 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.204294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bjq65" event={"ID":"2c77f4a4-6fbf-4ca3-8291-a48491c2f423","Type":"ContainerStarted","Data":"5296feaaf60c99138b21084aa48189861b95cae231a351ff8d26efbf044da085"}
Jul 07 14:58:27.204476 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.204368    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bjq65" event={"ID":"2c77f4a4-6fbf-4ca3-8291-a48491c2f423","Type":"ContainerStarted","Data":"fc1237cc652ce599101273c53073d21efa7cf93d14bfaf74d1aaf72302fee4ec"}
Jul 07 14:58:27.206923 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.206892    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx" event={"ID":"faad4c48-420e-4d7d-9673-1d63790e2c33","Type":"ContainerStarted","Data":"c72de58460c925dc2b9d8f30367fde4141943f12f04fa1b10ecc675cc43605c0"}
Jul 07 14:58:27.207045 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.206952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx" event={"ID":"faad4c48-420e-4d7d-9673-1d63790e2c33","Type":"ContainerStarted","Data":"ff1b094257ad1aae82bc1d1a6f61e0b89bd9fc80a6db439625e408cf363d537b"}
Jul 07 14:58:27.208268 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.208241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tq54m" event={"ID":"334b7bba-a8fa-4589-98db-4f59626c7da6","Type":"ContainerStarted","Data":"cce7485d9bfe9804bfd2aabf646d4d13d61da4ac1a2b8df2818e87571aac216e"}
Jul 07 14:58:27.227115 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.227050    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bjq65" podStartSLOduration=66.834785898 podStartE2EDuration="1m8.226992713s" totalImagesPullingTime="1.392206815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:27.226819313 +0000 UTC m=+68.959431635" watchObservedRunningTime="2026-07-07 14:58:27.226992713 +0000 UTC m=+68.959605034"
Jul 07 14:58:27.245008 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:27.244969    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:29.164719 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.164684    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gdcl2"
Jul 07 14:58:29.267703 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.267669    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-898cf4bf8-hnjwl"]
Jul 07 14:58:29.269810 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.269786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.273333 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.273307    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 07 14:58:29.273731 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.273707    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cprumjevf7bvg\""
Jul 07 14:58:29.273830 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.273775    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xxrq9\""
Jul 07 14:58:29.273892 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.273828    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 07 14:58:29.273941 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.273712    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 07 14:58:29.274857 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.274837    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 14:58:29.284173 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.284144    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-898cf4bf8-hnjwl"]
Jul 07 14:58:29.347793 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.347750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fbc65dc2-5780-48ac-82a6-47e65b143112-audit-log\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.347986 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.347802    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fbc65dc2-5780-48ac-82a6-47e65b143112-metrics-server-audit-profiles\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.347986 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.347872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fbc65dc2-5780-48ac-82a6-47e65b143112-secret-metrics-server-tls\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.347986 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.347902    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5tc5\" (UniqueName: \"kubernetes.io/projected/fbc65dc2-5780-48ac-82a6-47e65b143112-kube-api-access-m5tc5\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.347986 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.347927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fbc65dc2-5780-48ac-82a6-47e65b143112-secret-metrics-server-client-certs\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.347986 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.347960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fbc65dc2-5780-48ac-82a6-47e65b143112-client-ca-bundle\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.348233 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.348076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc65dc2-5780-48ac-82a6-47e65b143112-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.449084 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.448991    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fbc65dc2-5780-48ac-82a6-47e65b143112-secret-metrics-server-client-certs\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.449084 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.449035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fbc65dc2-5780-48ac-82a6-47e65b143112-client-ca-bundle\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.449324 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.449096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc65dc2-5780-48ac-82a6-47e65b143112-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.449324 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.449122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fbc65dc2-5780-48ac-82a6-47e65b143112-audit-log\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.449324 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.449148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fbc65dc2-5780-48ac-82a6-47e65b143112-metrics-server-audit-profiles\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.449324 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.449197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fbc65dc2-5780-48ac-82a6-47e65b143112-secret-metrics-server-tls\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.449324 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.449223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5tc5\" (UniqueName: \"kubernetes.io/projected/fbc65dc2-5780-48ac-82a6-47e65b143112-kube-api-access-m5tc5\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.449747 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.449695    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fbc65dc2-5780-48ac-82a6-47e65b143112-audit-log\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.450096 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.450062    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc65dc2-5780-48ac-82a6-47e65b143112-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.450452 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.450417    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fbc65dc2-5780-48ac-82a6-47e65b143112-metrics-server-audit-profiles\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.451951 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.451930    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fbc65dc2-5780-48ac-82a6-47e65b143112-secret-metrics-server-client-certs\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.452198 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.452176    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fbc65dc2-5780-48ac-82a6-47e65b143112-client-ca-bundle\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.452286 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.452181    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fbc65dc2-5780-48ac-82a6-47e65b143112-secret-metrics-server-tls\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.460292 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.460268    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5tc5\" (UniqueName: \"kubernetes.io/projected/fbc65dc2-5780-48ac-82a6-47e65b143112-kube-api-access-m5tc5\") pod \"metrics-server-898cf4bf8-hnjwl\" (UID: \"fbc65dc2-5780-48ac-82a6-47e65b143112\") " pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.581572 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.581523    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:29.729728 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.729644    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns"]
Jul 07 14:58:29.732514 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.732492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns"
Jul 07 14:58:29.735462 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.735430    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 07 14:58:29.735601 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.735579    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jpnf9\""
Jul 07 14:58:29.742821 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.742799    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns"]
Jul 07 14:58:29.852023 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.851979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c4cf86be-6802-4c24-92fd-49f31354feaf-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-zq9ns\" (UID: \"c4cf86be-6802-4c24-92fd-49f31354feaf\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns"
Jul 07 14:58:29.852205 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.852052    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6hpm\" (UniqueName: \"kubernetes.io/projected/c4cf86be-6802-4c24-92fd-49f31354feaf-kube-api-access-h6hpm\") pod \"monitoring-plugin-d5dcff7cf-zq9ns\" (UID: \"c4cf86be-6802-4c24-92fd-49f31354feaf\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns"
Jul 07 14:58:29.953482 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.953461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6hpm\" (UniqueName: \"kubernetes.io/projected/c4cf86be-6802-4c24-92fd-49f31354feaf-kube-api-access-h6hpm\") pod \"monitoring-plugin-d5dcff7cf-zq9ns\" (UID: \"c4cf86be-6802-4c24-92fd-49f31354feaf\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns"
Jul 07 14:58:29.953572 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.953546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c4cf86be-6802-4c24-92fd-49f31354feaf-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-zq9ns\" (UID: \"c4cf86be-6802-4c24-92fd-49f31354feaf\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns"
Jul 07 14:58:29.953669 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:29.953652    2581 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 07 14:58:29.953730 ip-10-0-143-168 kubenswrapper[2581]: E0707 14:58:29.953720    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4cf86be-6802-4c24-92fd-49f31354feaf-monitoring-plugin-cert podName:c4cf86be-6802-4c24-92fd-49f31354feaf nodeName:}" failed. No retries permitted until 2026-07-07 14:58:30.453699842 +0000 UTC m=+72.186312142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/c4cf86be-6802-4c24-92fd-49f31354feaf-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-zq9ns" (UID: "c4cf86be-6802-4c24-92fd-49f31354feaf") : secret "monitoring-plugin-cert" not found
Jul 07 14:58:29.962831 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:29.962805    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6hpm\" (UniqueName: \"kubernetes.io/projected/c4cf86be-6802-4c24-92fd-49f31354feaf-kube-api-access-h6hpm\") pod \"monitoring-plugin-d5dcff7cf-zq9ns\" (UID: \"c4cf86be-6802-4c24-92fd-49f31354feaf\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns"
Jul 07 14:58:30.333840 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:30.333735    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"]
Jul 07 14:58:30.425335 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:58:30.425295    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504ebd6e_ecc5_4a6a_9afc_59648f933fb4.slice/crio-9548c6f738e0b43f2081d1c04318607fe869ec34fddf3dff470c9af683d2b2ac WatchSource:0}: Error finding container 9548c6f738e0b43f2081d1c04318607fe869ec34fddf3dff470c9af683d2b2ac: Status 404 returned error can't find the container with id 9548c6f738e0b43f2081d1c04318607fe869ec34fddf3dff470c9af683d2b2ac
Jul 07 14:58:30.458923 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:30.458871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c4cf86be-6802-4c24-92fd-49f31354feaf-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-zq9ns\" (UID: \"c4cf86be-6802-4c24-92fd-49f31354feaf\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns"
Jul 07 14:58:30.462291 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:30.462266    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c4cf86be-6802-4c24-92fd-49f31354feaf-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-zq9ns\" (UID: \"c4cf86be-6802-4c24-92fd-49f31354feaf\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns"
Jul 07 14:58:30.605882 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:30.605851    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-898cf4bf8-hnjwl"]
Jul 07 14:58:30.609637 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:58:30.609603    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc65dc2_5780_48ac_82a6_47e65b143112.slice/crio-612e94665dd6786bc489bb3166c90af9048d20d2033a9bda829e36c65cb970c6 WatchSource:0}: Error finding container 612e94665dd6786bc489bb3166c90af9048d20d2033a9bda829e36c65cb970c6: Status 404 returned error can't find the container with id 612e94665dd6786bc489bb3166c90af9048d20d2033a9bda829e36c65cb970c6
Jul 07 14:58:30.645366 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:30.645309    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns"
Jul 07 14:58:30.822875 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:30.822836    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns"]
Jul 07 14:58:31.235493 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.235427    2581 generic.go:356] "Generic (PLEG): container finished" podID="329e599d-b769-4855-940e-2dbb2f95bf2b" containerID="5dbdd6722c75e090b6cde93306418539e7ea0e58e340dc8e31e6e738be3041f3" exitCode=0
Jul 07 14:58:31.235898 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.235539    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"329e599d-b769-4855-940e-2dbb2f95bf2b","Type":"ContainerDied","Data":"5dbdd6722c75e090b6cde93306418539e7ea0e58e340dc8e31e6e738be3041f3"}
Jul 07 14:58:31.242328 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.239423    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns" event={"ID":"c4cf86be-6802-4c24-92fd-49f31354feaf","Type":"ContainerStarted","Data":"35e0938afe9ea5ad2d64e39021eb130bb32fa2726ee8cf31ffdc30605f412d0a"}
Jul 07 14:58:31.243098 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.243044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl" event={"ID":"fbc65dc2-5780-48ac-82a6-47e65b143112","Type":"ContainerStarted","Data":"612e94665dd6786bc489bb3166c90af9048d20d2033a9bda829e36c65cb970c6"}
Jul 07 14:58:31.246522 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.246476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b9d6j" event={"ID":"386c6191-61a2-412c-9331-152a3d94baf9","Type":"ContainerStarted","Data":"99fd912cad23cb5e5894464ccf3ac71d4b73b84ec6ce34b4ed05778036e9ff7f"}
Jul 07 14:58:31.248326 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.248242    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk" event={"ID":"504ebd6e-ecc5-4a6a-9afc-59648f933fb4","Type":"ContainerStarted","Data":"9548c6f738e0b43f2081d1c04318607fe869ec34fddf3dff470c9af683d2b2ac"}
Jul 07 14:58:31.250782 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.250240    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qk5j7" event={"ID":"0043d8b0-5817-4b30-950b-0145e8690f3f","Type":"ContainerStarted","Data":"854aa9d3e3ab1d26cab93d7921b762a34fb1f7ae12308a1258198e7d270a7810"}
Jul 07 14:58:31.250782 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.250744    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:58:31.254048 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.254025    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n" event={"ID":"2e6d5ccf-86b8-49d2-9a73-835584742a65","Type":"ContainerStarted","Data":"92d09ffc9023a1980132278eba2660a1a0e08b34080fc4d07422bb05b5fcc8ab"}
Jul 07 14:58:31.254134 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.254085    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n" event={"ID":"2e6d5ccf-86b8-49d2-9a73-835584742a65","Type":"ContainerStarted","Data":"4704151780e9350564017dfc74d71fa366cba5c160ecb8d84d5668b8195841fb"}
Jul 07 14:58:31.254134 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.254118    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n" event={"ID":"2e6d5ccf-86b8-49d2-9a73-835584742a65","Type":"ContainerStarted","Data":"ce31b119299c27b0d39c9cc53337b27a29f88f8fef12e4e11ddf8bb441ee1e2c"}
Jul 07 14:58:31.257116 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.257058    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx" event={"ID":"faad4c48-420e-4d7d-9673-1d63790e2c33","Type":"ContainerStarted","Data":"20cffae8adbdcd3c3901b1da147ffdf3cc2cf5c5fba865b92855f7d5fa91eb54"}
Jul 07 14:58:31.259362 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.259263    2581 generic.go:356] "Generic (PLEG): container finished" podID="334b7bba-a8fa-4589-98db-4f59626c7da6" containerID="448404fd969e8300a467f5c5226d11b1970292e145fe4e5623d1d40077419b18" exitCode=0
Jul 07 14:58:31.259362 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.259291    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tq54m" event={"ID":"334b7bba-a8fa-4589-98db-4f59626c7da6","Type":"ContainerDied","Data":"448404fd969e8300a467f5c5226d11b1970292e145fe4e5623d1d40077419b18"}
Jul 07 14:58:31.351444 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.350701    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qk5j7" podStartSLOduration=66.723612383 podStartE2EDuration="1m12.350677776s" totalImagesPullingTime="5.627065393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:31.345259842 +0000 UTC m=+73.077872163" watchObservedRunningTime="2026-07-07 14:58:31.350677776 +0000 UTC m=+73.083290100"
Jul 07 14:58:31.354505 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.354478    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 14:58:31.364841 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.364814    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.373481 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.371153    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 14:58:31.373481 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.371592    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 14:58:31.373481 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.371799    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 14:58:31.373481 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.371963    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4grcv\""
Jul 07 14:58:31.373481 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.372105    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 14:58:31.373481 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.372157    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 14:58:31.373481 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.373366    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3huvrb3jh1qlc\""
Jul 07 14:58:31.374537 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.373613    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 14:58:31.374537 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.373674    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 14:58:31.374537 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.373740    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 14:58:31.374537 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.373799    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 14:58:31.374537 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.373946    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 14:58:31.374537 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.374004    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 14:58:31.374537 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.374054    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 14:58:31.385815 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.383947    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 14:58:31.385815 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.384764    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 14:58:31.388319 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.388271    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-5lx8n" podStartSLOduration=2.6269243319999998 podStartE2EDuration="7.388252744s" totalImagesPullingTime="4.761328412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:31.384439062 +0000 UTC m=+73.117051385" watchObservedRunningTime="2026-07-07 14:58:31.388252744 +0000 UTC m=+73.120865063"
Jul 07 14:58:31.391475 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.391455    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 14:58:31.410554 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.410494    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b9d6j" podStartSLOduration=66.767073592 podStartE2EDuration="1m12.410473259s" totalImagesPullingTime="5.643399667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:31.407858784 +0000 UTC m=+73.140471106" watchObservedRunningTime="2026-07-07 14:58:31.410473259 +0000 UTC m=+73.143085575"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-config\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6704654e-8d88-4f83-bac2-c55b696da1f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdjtb\" (UniqueName: \"kubernetes.io/projected/6704654e-8d88-4f83-bac2-c55b696da1f6-kube-api-access-sdjtb\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6704654e-8d88-4f83-bac2-c55b696da1f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.470899 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.471797 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.471797 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.471797 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.471797 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.470637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6704654e-8d88-4f83-bac2-c55b696da1f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6704654e-8d88-4f83-bac2-c55b696da1f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6704654e-8d88-4f83-bac2-c55b696da1f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-config\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.573459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.574512 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6704654e-8d88-4f83-bac2-c55b696da1f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.574512 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.574512 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.572453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdjtb\" (UniqueName: \"kubernetes.io/projected/6704654e-8d88-4f83-bac2-c55b696da1f6-kube-api-access-sdjtb\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.574512 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.573315    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.576259 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.574991    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.576259 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.575308    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6704654e-8d88-4f83-bac2-c55b696da1f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.576259 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.575660    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.576259 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.575970    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.576259 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.575985    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.577149 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.577122    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.579451 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.579428    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6704654e-8d88-4f83-bac2-c55b696da1f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.585216 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.585186    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.585335 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.585275    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdjtb\" (UniqueName: \"kubernetes.io/projected/6704654e-8d88-4f83-bac2-c55b696da1f6-kube-api-access-sdjtb\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.596045 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.595980    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6704654e-8d88-4f83-bac2-c55b696da1f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.596045 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.596004    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.596045 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.596009    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.596585 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.596449    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.596585 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.596532    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-config\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.596877 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.596854    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.597242 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.597180    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.597845 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.597419    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.597845 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.597716    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6704654e-8d88-4f83-bac2-c55b696da1f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.597845 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.597768    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6704654e-8d88-4f83-bac2-c55b696da1f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6704654e-8d88-4f83-bac2-c55b696da1f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.687759 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.687720    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:31.969004 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:31.968889    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 14:58:31.978224 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:58:31.978167    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6704654e_8d88_4f83_bac2_c55b696da1f6.slice/crio-a6b9ef438720f09d24bae00d44141acb99aa22fc1bcf65b071e89c161f801b7c WatchSource:0}: Error finding container a6b9ef438720f09d24bae00d44141acb99aa22fc1bcf65b071e89c161f801b7c: Status 404 returned error can't find the container with id a6b9ef438720f09d24bae00d44141acb99aa22fc1bcf65b071e89c161f801b7c
Jul 07 14:58:32.266966 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:32.266885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx" event={"ID":"faad4c48-420e-4d7d-9673-1d63790e2c33","Type":"ContainerStarted","Data":"6bb8a813cdb609713725a2cce1d8f71546e02d743251dcc58ce8b253bf778b63"}
Jul 07 14:58:32.269639 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:32.269600    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6704654e-8d88-4f83-bac2-c55b696da1f6","Type":"ContainerStarted","Data":"f117e3453bd297a79bdd088214c94679194267b9035783a1e2ff8a9652ca4d58"}
Jul 07 14:58:32.269777 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:32.269684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6704654e-8d88-4f83-bac2-c55b696da1f6","Type":"ContainerStarted","Data":"a6b9ef438720f09d24bae00d44141acb99aa22fc1bcf65b071e89c161f801b7c"}
Jul 07 14:58:32.271954 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:32.271924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tq54m" event={"ID":"334b7bba-a8fa-4589-98db-4f59626c7da6","Type":"ContainerStarted","Data":"ef5c85ef2a0ecf405807245c6ec989b43c59b7d3ef325941a6d4f2bb98881295"}
Jul 07 14:58:32.272052 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:32.271971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tq54m" event={"ID":"334b7bba-a8fa-4589-98db-4f59626c7da6","Type":"ContainerStarted","Data":"b0d249403e0c16f087cbf425a4b6eb4015b97925446fbf8bef69fcf06c7aa812"}
Jul 07 14:58:32.339385 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:32.338791    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xbxlx" podStartSLOduration=7.082464429 podStartE2EDuration="8.338769041s" totalImagesPullingTime="1.256304612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:32.337993219 +0000 UTC m=+74.070605540" watchObservedRunningTime="2026-07-07 14:58:32.338769041 +0000 UTC m=+74.071381364"
Jul 07 14:58:32.417194 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:32.417087    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tq54m" podStartSLOduration=4.668963806 podStartE2EDuration="8.417066413s" totalImagesPullingTime="3.657303608s" totalInitContainerRuntime="90.798999ms" isStatefulPod=true podCreationTimestamp="2026-07-07 14:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:32.415027785 +0000 UTC m=+74.147640107" watchObservedRunningTime="2026-07-07 14:58:32.417066413 +0000 UTC m=+74.149678716"
Jul 07 14:58:33.277008 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:33.276974    2581 generic.go:356] "Generic (PLEG): container finished" podID="6704654e-8d88-4f83-bac2-c55b696da1f6" containerID="f117e3453bd297a79bdd088214c94679194267b9035783a1e2ff8a9652ca4d58" exitCode=0
Jul 07 14:58:33.277207 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:33.277090    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6704654e-8d88-4f83-bac2-c55b696da1f6","Type":"ContainerDied","Data":"f117e3453bd297a79bdd088214c94679194267b9035783a1e2ff8a9652ca4d58"}
Jul 07 14:58:34.286067 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:34.286029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"329e599d-b769-4855-940e-2dbb2f95bf2b","Type":"ContainerStarted","Data":"b2b9e1342a492339df7801138f4bf7bee77e80e5bfd6c6cf777b7bd9aeadd3be"}
Jul 07 14:58:34.286549 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:34.286150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"329e599d-b769-4855-940e-2dbb2f95bf2b","Type":"ContainerStarted","Data":"af248bb372f285bcc131cd9e76447e5bd4c5bd8abc5df2829e451d26139a76df"}
Jul 07 14:58:34.286549 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:34.286193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"329e599d-b769-4855-940e-2dbb2f95bf2b","Type":"ContainerStarted","Data":"a9b62e50cff11c1e75a17b690cc90c15d01a00c018d6afffe00ff1656b95a7d8"}
Jul 07 14:58:34.286549 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:34.286231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"329e599d-b769-4855-940e-2dbb2f95bf2b","Type":"ContainerStarted","Data":"99d7b4a3ab6d5860343f369787d698ccfb6d7de7b35e5fe6639594ee10fe0423"}
Jul 07 14:58:34.286549 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:34.286269    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"329e599d-b769-4855-940e-2dbb2f95bf2b","Type":"ContainerStarted","Data":"1163ad9487c22ef91e7099adb4947a006f0688eec3abfc6c3e47b3c946a1be75"}
Jul 07 14:58:34.288398 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:34.287967    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns" event={"ID":"c4cf86be-6802-4c24-92fd-49f31354feaf","Type":"ContainerStarted","Data":"c1c3ba33c26996e026f3375ee14e275efb00a0be161bf4d53172a4898538d8ab"}
Jul 07 14:58:34.288398 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:34.288316    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns"
Jul 07 14:58:34.289940 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:34.289837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl" event={"ID":"fbc65dc2-5780-48ac-82a6-47e65b143112","Type":"ContainerStarted","Data":"2fd4ae1f5b9253aa88b774096703c8ab8a95642434ed6615ca1555a38b1f696d"}
Jul 07 14:58:34.292815 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:34.292788    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk" event={"ID":"504ebd6e-ecc5-4a6a-9afc-59648f933fb4","Type":"ContainerStarted","Data":"e482897d5f800eaab5a3458b7b9590ec405743da76ee129f91c50e9ed163fd6c"}
Jul 07 14:58:34.292912 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:34.292858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk" event={"ID":"504ebd6e-ecc5-4a6a-9afc-59648f933fb4","Type":"ContainerStarted","Data":"98afa3966a6bac25b978ad1e8c67158de86081d99f6507229bf45c484eb9659c"}
Jul 07 14:58:34.292912 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:34.292900    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk" event={"ID":"504ebd6e-ecc5-4a6a-9afc-59648f933fb4","Type":"ContainerStarted","Data":"b8c4b28bb80ca168c6dd8e33bcac9d672191bd92eb586fb8bdb242d38829ce08"}
Jul 07 14:58:34.294736 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:34.294711    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns"
Jul 07 14:58:34.309116 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:34.309073    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zq9ns" podStartSLOduration=2.299775861 podStartE2EDuration="5.309059494s" totalImagesPullingTime="3.009283633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:34.30797413 +0000 UTC m=+76.040586476" watchObservedRunningTime="2026-07-07 14:58:34.309059494 +0000 UTC m=+76.041671791"
Jul 07 14:58:34.327459 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:34.327373    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl" podStartSLOduration=2.102607914 podStartE2EDuration="5.327332019s" totalImagesPullingTime="3.224724105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:34.325800624 +0000 UTC m=+76.058412948" watchObservedRunningTime="2026-07-07 14:58:34.327332019 +0000 UTC m=+76.059944339"
Jul 07 14:58:35.299926 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:35.299834    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"329e599d-b769-4855-940e-2dbb2f95bf2b","Type":"ContainerStarted","Data":"2d576f6ecfbf86e590de1cd418c700da189b36852292a98ffdb46c2036f7ea1f"}
Jul 07 14:58:35.302699 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:35.302673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk" event={"ID":"504ebd6e-ecc5-4a6a-9afc-59648f933fb4","Type":"ContainerStarted","Data":"1c36f7041f9628910bea8f6deae0222f889b6564989793e6f95d0d42c90fa87b"}
Jul 07 14:58:35.302809 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:35.302730    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk" event={"ID":"504ebd6e-ecc5-4a6a-9afc-59648f933fb4","Type":"ContainerStarted","Data":"298d2b9fbd777323a842ce5b286d62a524330072d08f58386e95a7b5fd748150"}
Jul 07 14:58:35.302809 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:35.302760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk" event={"ID":"504ebd6e-ecc5-4a6a-9afc-59648f933fb4","Type":"ContainerStarted","Data":"a58ae69b58f385e07013d13930bc47cdddc7a36d6f74887c74f3b8f703f957d1"}
Jul 07 14:58:35.330341 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:35.330286    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.083367228 podStartE2EDuration="10.330265187s" totalImagesPullingTime="7.218464335s" totalInitContainerRuntime="28.433624ms" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:35.327458678 +0000 UTC m=+77.060070999" watchObservedRunningTime="2026-07-07 14:58:35.330265187 +0000 UTC m=+77.062877508"
Jul 07 14:58:35.356056 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:35.355736    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk" podStartSLOduration=5.090948862 podStartE2EDuration="9.355710036s" totalImagesPullingTime="4.264761174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:35.353779049 +0000 UTC m=+77.086391406" watchObservedRunningTime="2026-07-07 14:58:35.355710036 +0000 UTC m=+77.088322357"
Jul 07 14:58:36.310695 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:36.310658    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:37.317373 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:37.317292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6704654e-8d88-4f83-bac2-c55b696da1f6","Type":"ContainerStarted","Data":"bc60a6998929ad3ee9d4c2ee24f27605ad5457544c38b90e27304a8858dc067d"}
Jul 07 14:58:37.317767 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:37.317396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6704654e-8d88-4f83-bac2-c55b696da1f6","Type":"ContainerStarted","Data":"2e255d86d484008b9720a1ffb7c8528ae90b8e47156ac432792a42ba863e026d"}
Jul 07 14:58:37.317767 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:37.317445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6704654e-8d88-4f83-bac2-c55b696da1f6","Type":"ContainerStarted","Data":"c1f8512d64f160e512fd64ce40f6b8bbd44356ccb0b5cce8f5704817d7517aaf"}
Jul 07 14:58:37.317767 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:37.317486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6704654e-8d88-4f83-bac2-c55b696da1f6","Type":"ContainerStarted","Data":"cb4e838c1439d0c4c6dd38dcafe7f3590a3ea7adc8fdba916d631a0aeb702da9"}
Jul 07 14:58:38.324413 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:38.324377    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6704654e-8d88-4f83-bac2-c55b696da1f6","Type":"ContainerStarted","Data":"3aea27376bcbe25a730a5e06d23edbe69532df20183a4188b458b2e031ceafb4"}
Jul 07 14:58:38.324776 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:38.324443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6704654e-8d88-4f83-bac2-c55b696da1f6","Type":"ContainerStarted","Data":"7088c7c21d2b6fb8e9c6db7ee63e5c3f9f3f185e067df0a9334727141f5d3487"}
Jul 07 14:58:38.366893 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:38.366841    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.605373336 podStartE2EDuration="7.366825503s" totalImagesPullingTime="3.733727911s" totalInitContainerRuntime="27.724256ms" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:38.365952526 +0000 UTC m=+80.098564874" watchObservedRunningTime="2026-07-07 14:58:38.366825503 +0000 UTC m=+80.099437864"
Jul 07 14:58:41.688626 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:41.688512    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:58:42.325426 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:42.325400    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f6f9bc8bb-lspfk"
Jul 07 14:58:44.710826 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:44.710792    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lmjpm_3bf7e0cf-669e-42c4-8ee8-76764abc256a/serve-healthcheck-canary/0.log"
Jul 07 14:58:49.582372 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:49.582316    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:58:49.582839 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:58:49.582392    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:59:03.280640 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:03.280610    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qk5j7"
Jul 07 14:59:09.587394 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:09.587365    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:59:09.591215 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:09.591191    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-898cf4bf8-hnjwl"
Jul 07 14:59:31.688685 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:31.688643    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:59:31.707851 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:31.707823    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:59:32.561167 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:32.561130    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 14:59:49.335553 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.335512    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-55856b8684-c4p2j"]
Jul 07 14:59:49.338636 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.338608    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.341714 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.341688    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 07 14:59:49.341913 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.341893    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 07 14:59:49.341978 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.341899    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 07 14:59:49.341978 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.341937    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 07 14:59:49.341978 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.341942    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 07 14:59:49.342130 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.342013    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8j94m\""
Jul 07 14:59:49.347974 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.347860    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 07 14:59:49.351720 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.351697    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55856b8684-c4p2j"]
Jul 07 14:59:49.454828 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.454793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87hrr\" (UniqueName: \"kubernetes.io/projected/10799fc0-1122-4432-b46e-13894dd6d7db-kube-api-access-87hrr\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.455031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.454834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10799fc0-1122-4432-b46e-13894dd6d7db-serving-certs-ca-bundle\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.455031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.454860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/10799fc0-1122-4432-b46e-13894dd6d7db-federate-client-tls\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.455031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.454887    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/10799fc0-1122-4432-b46e-13894dd6d7db-secret-telemeter-client\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.455031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.454933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10799fc0-1122-4432-b46e-13894dd6d7db-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.455031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.454991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/10799fc0-1122-4432-b46e-13894dd6d7db-telemeter-client-tls\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.455031 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.455010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10799fc0-1122-4432-b46e-13894dd6d7db-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.455229 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.455095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10799fc0-1122-4432-b46e-13894dd6d7db-metrics-client-ca\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.555822 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.555783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/10799fc0-1122-4432-b46e-13894dd6d7db-telemeter-client-tls\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.556032 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.555827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10799fc0-1122-4432-b46e-13894dd6d7db-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.556032 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.556007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10799fc0-1122-4432-b46e-13894dd6d7db-metrics-client-ca\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.556163 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.556049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87hrr\" (UniqueName: \"kubernetes.io/projected/10799fc0-1122-4432-b46e-13894dd6d7db-kube-api-access-87hrr\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.556163 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.556078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10799fc0-1122-4432-b46e-13894dd6d7db-serving-certs-ca-bundle\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.556163 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.556098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/10799fc0-1122-4432-b46e-13894dd6d7db-federate-client-tls\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.556163 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.556131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/10799fc0-1122-4432-b46e-13894dd6d7db-secret-telemeter-client\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.556163 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.556154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10799fc0-1122-4432-b46e-13894dd6d7db-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.556868 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.556841    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10799fc0-1122-4432-b46e-13894dd6d7db-metrics-client-ca\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.557064 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.556843    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10799fc0-1122-4432-b46e-13894dd6d7db-serving-certs-ca-bundle\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.557181 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.557138    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10799fc0-1122-4432-b46e-13894dd6d7db-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.558417 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.558394    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/10799fc0-1122-4432-b46e-13894dd6d7db-telemeter-client-tls\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.558667 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.558640    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/10799fc0-1122-4432-b46e-13894dd6d7db-secret-telemeter-client\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.558667 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.558663    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/10799fc0-1122-4432-b46e-13894dd6d7db-federate-client-tls\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.558786 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.558714    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10799fc0-1122-4432-b46e-13894dd6d7db-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.565133 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.565111    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87hrr\" (UniqueName: \"kubernetes.io/projected/10799fc0-1122-4432-b46e-13894dd6d7db-kube-api-access-87hrr\") pod \"telemeter-client-55856b8684-c4p2j\" (UID: \"10799fc0-1122-4432-b46e-13894dd6d7db\") " pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.651213 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.651123    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j"
Jul 07 14:59:49.801626 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:49.801595    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55856b8684-c4p2j"]
Jul 07 14:59:49.804214 ip-10-0-143-168 kubenswrapper[2581]: W0707 14:59:49.804182    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10799fc0_1122_4432_b46e_13894dd6d7db.slice/crio-f625b697b729c608e24b9853b5bdcbc561fe23e15fe8e522f78cfd0f63649432 WatchSource:0}: Error finding container f625b697b729c608e24b9853b5bdcbc561fe23e15fe8e522f78cfd0f63649432: Status 404 returned error can't find the container with id f625b697b729c608e24b9853b5bdcbc561fe23e15fe8e522f78cfd0f63649432
Jul 07 14:59:50.616627 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:50.616589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j" event={"ID":"10799fc0-1122-4432-b46e-13894dd6d7db","Type":"ContainerStarted","Data":"f625b697b729c608e24b9853b5bdcbc561fe23e15fe8e522f78cfd0f63649432"}
Jul 07 14:59:52.626650 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:52.626607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j" event={"ID":"10799fc0-1122-4432-b46e-13894dd6d7db","Type":"ContainerStarted","Data":"e03942b8607e500aa700d71a147740bd0bdb94e8438a08eca7e45ac0bae8f026"}
Jul 07 14:59:52.627143 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:52.626718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j" event={"ID":"10799fc0-1122-4432-b46e-13894dd6d7db","Type":"ContainerStarted","Data":"7eee23f56e3f3cbc005f7842f4c5ba9b8fafbedf96e2e52a7fff8f0268aba57e"}
Jul 07 14:59:52.627143 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:52.626751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j" event={"ID":"10799fc0-1122-4432-b46e-13894dd6d7db","Type":"ContainerStarted","Data":"e4443033031471e88c54e2a86a255c909795d20c7b0df995fcd8f3c31dbd23ca"}
Jul 07 14:59:52.652342 ip-10-0-143-168 kubenswrapper[2581]: I0707 14:59:52.652276    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-55856b8684-c4p2j" podStartSLOduration=1.651927393 podStartE2EDuration="3.652256351s" totalImagesPullingTime="2.000328958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:59:52.650576035 +0000 UTC m=+154.383188373" watchObservedRunningTime="2026-07-07 14:59:52.652256351 +0000 UTC m=+154.384868672"
Jul 07 15:02:18.695991 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:02:18.695959    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 15:05:05.438065 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.437981    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-b8dcz"]
Jul 07 15:05:05.440458 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.440436    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-b8dcz"
Jul 07 15:05:05.447208 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.447183    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 07 15:05:05.447319 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.447236    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 15:05:05.448662 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.448643    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 15:05:05.448758 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.448684    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-mcj5h\""
Jul 07 15:05:05.466903 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.466875    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-b8dcz"]
Jul 07 15:05:05.540711 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.540672    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkjxb\" (UniqueName: \"kubernetes.io/projected/ccc1e2f1-8be8-4f60-ac53-5ca78eda47d2-kube-api-access-vkjxb\") pod \"odh-model-controller-84b54b7f97-b8dcz\" (UID: \"ccc1e2f1-8be8-4f60-ac53-5ca78eda47d2\") " pod="kserve/odh-model-controller-84b54b7f97-b8dcz"
Jul 07 15:05:05.540875 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.540736    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc1e2f1-8be8-4f60-ac53-5ca78eda47d2-cert\") pod \"odh-model-controller-84b54b7f97-b8dcz\" (UID: \"ccc1e2f1-8be8-4f60-ac53-5ca78eda47d2\") " pod="kserve/odh-model-controller-84b54b7f97-b8dcz"
Jul 07 15:05:05.642109 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.642071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkjxb\" (UniqueName: \"kubernetes.io/projected/ccc1e2f1-8be8-4f60-ac53-5ca78eda47d2-kube-api-access-vkjxb\") pod \"odh-model-controller-84b54b7f97-b8dcz\" (UID: \"ccc1e2f1-8be8-4f60-ac53-5ca78eda47d2\") " pod="kserve/odh-model-controller-84b54b7f97-b8dcz"
Jul 07 15:05:05.642282 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.642138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc1e2f1-8be8-4f60-ac53-5ca78eda47d2-cert\") pod \"odh-model-controller-84b54b7f97-b8dcz\" (UID: \"ccc1e2f1-8be8-4f60-ac53-5ca78eda47d2\") " pod="kserve/odh-model-controller-84b54b7f97-b8dcz"
Jul 07 15:05:05.644646 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.644625    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc1e2f1-8be8-4f60-ac53-5ca78eda47d2-cert\") pod \"odh-model-controller-84b54b7f97-b8dcz\" (UID: \"ccc1e2f1-8be8-4f60-ac53-5ca78eda47d2\") " pod="kserve/odh-model-controller-84b54b7f97-b8dcz"
Jul 07 15:05:05.652311 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.652289    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkjxb\" (UniqueName: \"kubernetes.io/projected/ccc1e2f1-8be8-4f60-ac53-5ca78eda47d2-kube-api-access-vkjxb\") pod \"odh-model-controller-84b54b7f97-b8dcz\" (UID: \"ccc1e2f1-8be8-4f60-ac53-5ca78eda47d2\") " pod="kserve/odh-model-controller-84b54b7f97-b8dcz"
Jul 07 15:05:05.750714 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.750611    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-b8dcz"
Jul 07 15:05:05.899756 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.899726    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-b8dcz"]
Jul 07 15:05:05.903192 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:05:05.903166    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc1e2f1_8be8_4f60_ac53_5ca78eda47d2.slice/crio-826b23ecb4bfe2590b0df78e4d2ac41a572fee3e18305a503a17f4d19f9435dc WatchSource:0}: Error finding container 826b23ecb4bfe2590b0df78e4d2ac41a572fee3e18305a503a17f4d19f9435dc: Status 404 returned error can't find the container with id 826b23ecb4bfe2590b0df78e4d2ac41a572fee3e18305a503a17f4d19f9435dc
Jul 07 15:05:05.904875 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:05.904850    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 15:05:06.857920 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:06.857884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-b8dcz" event={"ID":"ccc1e2f1-8be8-4f60-ac53-5ca78eda47d2","Type":"ContainerStarted","Data":"826b23ecb4bfe2590b0df78e4d2ac41a572fee3e18305a503a17f4d19f9435dc"}
Jul 07 15:05:08.868625 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:08.868596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-b8dcz" event={"ID":"ccc1e2f1-8be8-4f60-ac53-5ca78eda47d2","Type":"ContainerStarted","Data":"4b938e61234b77134dea3054a0495154dc197c1905c88833552f784023e38423"}
Jul 07 15:05:08.869039 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:08.868756    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-b8dcz"
Jul 07 15:05:08.895489 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:08.895440    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-b8dcz" podStartSLOduration=1.051506842 podStartE2EDuration="3.895424999s" totalImagesPullingTime="2.843918157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:05:08.891671197 +0000 UTC m=+470.624283520" watchObservedRunningTime="2026-07-07 15:05:08.895424999 +0000 UTC m=+470.628037348"
Jul 07 15:05:19.875313 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:19.875282    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-b8dcz"
Jul 07 15:05:20.753699 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:20.753667    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-k7q4c"]
Jul 07 15:05:20.755973 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:20.755956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-k7q4c"
Jul 07 15:05:20.762244 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:20.761994    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 15:05:20.762604 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:20.762577    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-xkrnz\""
Jul 07 15:05:20.774772 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:20.774750    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-k7q4c"]
Jul 07 15:05:20.886773 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:20.886732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c44693e-46aa-48c6-b326-f57b52e70db1-model-cache\") pod \"s3-init-k7q4c\" (UID: \"2c44693e-46aa-48c6-b326-f57b52e70db1\") " pod="kserve/s3-init-k7q4c"
Jul 07 15:05:20.887170 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:20.886794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kh54\" (UniqueName: \"kubernetes.io/projected/2c44693e-46aa-48c6-b326-f57b52e70db1-kube-api-access-5kh54\") pod \"s3-init-k7q4c\" (UID: \"2c44693e-46aa-48c6-b326-f57b52e70db1\") " pod="kserve/s3-init-k7q4c"
Jul 07 15:05:20.987334 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:20.987293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c44693e-46aa-48c6-b326-f57b52e70db1-model-cache\") pod \"s3-init-k7q4c\" (UID: \"2c44693e-46aa-48c6-b326-f57b52e70db1\") " pod="kserve/s3-init-k7q4c"
Jul 07 15:05:20.987529 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:20.987381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kh54\" (UniqueName: \"kubernetes.io/projected/2c44693e-46aa-48c6-b326-f57b52e70db1-kube-api-access-5kh54\") pod \"s3-init-k7q4c\" (UID: \"2c44693e-46aa-48c6-b326-f57b52e70db1\") " pod="kserve/s3-init-k7q4c"
Jul 07 15:05:20.987789 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:20.987769    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c44693e-46aa-48c6-b326-f57b52e70db1-model-cache\") pod \"s3-init-k7q4c\" (UID: \"2c44693e-46aa-48c6-b326-f57b52e70db1\") " pod="kserve/s3-init-k7q4c"
Jul 07 15:05:20.998721 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:20.998700    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kh54\" (UniqueName: \"kubernetes.io/projected/2c44693e-46aa-48c6-b326-f57b52e70db1-kube-api-access-5kh54\") pod \"s3-init-k7q4c\" (UID: \"2c44693e-46aa-48c6-b326-f57b52e70db1\") " pod="kserve/s3-init-k7q4c"
Jul 07 15:05:21.087546 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:21.087461    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-k7q4c"
Jul 07 15:05:21.223970 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:21.223943    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-k7q4c"]
Jul 07 15:05:21.225610 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:05:21.225580    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c44693e_46aa_48c6_b326_f57b52e70db1.slice/crio-c7104703456a4d4ba8e681c446b6bcf0ae88ad456d8df4f62d1f00d8afa54d2e WatchSource:0}: Error finding container c7104703456a4d4ba8e681c446b6bcf0ae88ad456d8df4f62d1f00d8afa54d2e: Status 404 returned error can't find the container with id c7104703456a4d4ba8e681c446b6bcf0ae88ad456d8df4f62d1f00d8afa54d2e
Jul 07 15:05:21.921701 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:21.921666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-k7q4c" event={"ID":"2c44693e-46aa-48c6-b326-f57b52e70db1","Type":"ContainerStarted","Data":"c7104703456a4d4ba8e681c446b6bcf0ae88ad456d8df4f62d1f00d8afa54d2e"}
Jul 07 15:05:25.941121 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:25.941082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-k7q4c" event={"ID":"2c44693e-46aa-48c6-b326-f57b52e70db1","Type":"ContainerStarted","Data":"0cbda3157f47ab116d95f0d0534fd7900377a583e07a47dc35f350a277474eb9"}
Jul 07 15:05:27.950387 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:27.950327    2581 generic.go:356] "Generic (PLEG): container finished" podID="2c44693e-46aa-48c6-b326-f57b52e70db1" containerID="0cbda3157f47ab116d95f0d0534fd7900377a583e07a47dc35f350a277474eb9" exitCode=0
Jul 07 15:05:27.950870 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:27.950406    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-k7q4c" event={"ID":"2c44693e-46aa-48c6-b326-f57b52e70db1","Type":"ContainerDied","Data":"0cbda3157f47ab116d95f0d0534fd7900377a583e07a47dc35f350a277474eb9"}
Jul 07 15:05:32.976302 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:32.976213    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-k7q4c" event={"ID":"2c44693e-46aa-48c6-b326-f57b52e70db1","Type":"ContainerStarted","Data":"9d4bd0fde0b44c1af308eeb736025fd4c7a44e0a4a39c11c0a411a57767eda11"}
Jul 07 15:05:32.999708 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:05:32.999662    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-k7q4c" podStartSLOduration=1.909466536 podStartE2EDuration="12.999646952s" totalImagesPullingTime="8.724183906s" totalInitContainerRuntime="2.36599651s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:05:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:05:32.99798332 +0000 UTC m=+494.730595652" watchObservedRunningTime="2026-07-07 15:05:32.999646952 +0000 UTC m=+494.732259271"
Jul 07 15:06:38.247217 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:38.247178    2581 generic.go:356] "Generic (PLEG): container finished" podID="2c44693e-46aa-48c6-b326-f57b52e70db1" containerID="9d4bd0fde0b44c1af308eeb736025fd4c7a44e0a4a39c11c0a411a57767eda11" exitCode=0
Jul 07 15:06:38.247648 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:38.247226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-k7q4c" event={"ID":"2c44693e-46aa-48c6-b326-f57b52e70db1","Type":"ContainerDied","Data":"9d4bd0fde0b44c1af308eeb736025fd4c7a44e0a4a39c11c0a411a57767eda11"}
Jul 07 15:06:39.377175 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:39.377151    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-k7q4c"
Jul 07 15:06:39.482084 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:39.482043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kh54\" (UniqueName: \"kubernetes.io/projected/2c44693e-46aa-48c6-b326-f57b52e70db1-kube-api-access-5kh54\") pod \"2c44693e-46aa-48c6-b326-f57b52e70db1\" (UID: \"2c44693e-46aa-48c6-b326-f57b52e70db1\") "
Jul 07 15:06:39.482084 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:39.482087    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c44693e-46aa-48c6-b326-f57b52e70db1-model-cache\") pod \"2c44693e-46aa-48c6-b326-f57b52e70db1\" (UID: \"2c44693e-46aa-48c6-b326-f57b52e70db1\") "
Jul 07 15:06:39.484356 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:39.484320    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c44693e-46aa-48c6-b326-f57b52e70db1-kube-api-access-5kh54" (OuterVolumeSpecName: "kube-api-access-5kh54") pod "2c44693e-46aa-48c6-b326-f57b52e70db1" (UID: "2c44693e-46aa-48c6-b326-f57b52e70db1"). InnerVolumeSpecName "kube-api-access-5kh54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:06:39.500819 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:39.500778    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c44693e-46aa-48c6-b326-f57b52e70db1-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c44693e-46aa-48c6-b326-f57b52e70db1" (UID: "2c44693e-46aa-48c6-b326-f57b52e70db1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:06:39.582695 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:39.582605    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kh54\" (UniqueName: \"kubernetes.io/projected/2c44693e-46aa-48c6-b326-f57b52e70db1-kube-api-access-5kh54\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:06:39.582695 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:39.582638    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c44693e-46aa-48c6-b326-f57b52e70db1-model-cache\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:06:40.257894 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:40.257863    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-k7q4c"
Jul 07 15:06:40.258069 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:40.257866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-k7q4c" event={"ID":"2c44693e-46aa-48c6-b326-f57b52e70db1","Type":"ContainerDied","Data":"c7104703456a4d4ba8e681c446b6bcf0ae88ad456d8df4f62d1f00d8afa54d2e"}
Jul 07 15:06:40.258069 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:40.257975    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7104703456a4d4ba8e681c446b6bcf0ae88ad456d8df4f62d1f00d8afa54d2e"
Jul 07 15:06:52.070479 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.070440    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"]
Jul 07 15:06:52.070951 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.070772    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c44693e-46aa-48c6-b326-f57b52e70db1" containerName="s3-init"
Jul 07 15:06:52.070951 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.070783    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c44693e-46aa-48c6-b326-f57b52e70db1" containerName="s3-init"
Jul 07 15:06:52.070951 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.070801    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c44693e-46aa-48c6-b326-f57b52e70db1" containerName="download-hf-model"
Jul 07 15:06:52.070951 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.070807    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c44693e-46aa-48c6-b326-f57b52e70db1" containerName="download-hf-model"
Jul 07 15:06:52.070951 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.070867    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c44693e-46aa-48c6-b326-f57b52e70db1" containerName="s3-init"
Jul 07 15:06:52.072963 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.072947    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.078207 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.078184    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 15:06:52.078359 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.078184    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 15:06:52.078359 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.078184    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-df08e-kube-rbac-proxy-sar-config\""
Jul 07 15:06:52.078594 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.078498    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z5ftp\""
Jul 07 15:06:52.079822 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.079656    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 15:06:52.080007 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.079964    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-df08e-predictor-serving-cert\""
Jul 07 15:06:52.080007 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.079989    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 15:06:52.097231 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.097205    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"]
Jul 07 15:06:52.193251 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.193207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eec569-d4d4-406f-8927-f67a43479ae6-proxy-tls\") pod \"isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.193489 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.193307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6ss9\" (UniqueName: \"kubernetes.io/projected/91eec569-d4d4-406f-8927-f67a43479ae6-kube-api-access-h6ss9\") pod \"isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.193489 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.193364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91eec569-d4d4-406f-8927-f67a43479ae6-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.193489 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.193397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91eec569-d4d4-406f-8927-f67a43479ae6-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.193489 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.193447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91eec569-d4d4-406f-8927-f67a43479ae6-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.293889 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.293856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91eec569-d4d4-406f-8927-f67a43479ae6-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.294082 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.293906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91eec569-d4d4-406f-8927-f67a43479ae6-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.294082 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.293954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eec569-d4d4-406f-8927-f67a43479ae6-proxy-tls\") pod \"isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.294082 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.294013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6ss9\" (UniqueName: \"kubernetes.io/projected/91eec569-d4d4-406f-8927-f67a43479ae6-kube-api-access-h6ss9\") pod \"isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.294251 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.294196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91eec569-d4d4-406f-8927-f67a43479ae6-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.294340 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.294316    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91eec569-d4d4-406f-8927-f67a43479ae6-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.294727 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.294705    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91eec569-d4d4-406f-8927-f67a43479ae6-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.294815 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.294773    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91eec569-d4d4-406f-8927-f67a43479ae6-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.296490 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.296471    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eec569-d4d4-406f-8927-f67a43479ae6-proxy-tls\") pod \"isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.303394 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.303372    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6ss9\" (UniqueName: \"kubernetes.io/projected/91eec569-d4d4-406f-8927-f67a43479ae6-kube-api-access-h6ss9\") pod \"isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.383560 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.383465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:06:52.524206 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:52.524113    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"]
Jul 07 15:06:52.526628 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:06:52.526596    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91eec569_d4d4_406f_8927_f67a43479ae6.slice/crio-d96ac4f2ba6e7455b2d696dd0254c73db05464715e2b044703e686b159f843b7 WatchSource:0}: Error finding container d96ac4f2ba6e7455b2d696dd0254c73db05464715e2b044703e686b159f843b7: Status 404 returned error can't find the container with id d96ac4f2ba6e7455b2d696dd0254c73db05464715e2b044703e686b159f843b7
Jul 07 15:06:53.312688 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:53.312656    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" event={"ID":"91eec569-d4d4-406f-8927-f67a43479ae6","Type":"ContainerStarted","Data":"dbd4b8a3e55244f74784973c48de473d82e0e06b508ffac75690df7225281ea7"}
Jul 07 15:06:53.313100 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:53.312723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" event={"ID":"91eec569-d4d4-406f-8927-f67a43479ae6","Type":"ContainerStarted","Data":"d96ac4f2ba6e7455b2d696dd0254c73db05464715e2b044703e686b159f843b7"}
Jul 07 15:06:56.333864 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:56.333830    2581 generic.go:356] "Generic (PLEG): container finished" podID="91eec569-d4d4-406f-8927-f67a43479ae6" containerID="dbd4b8a3e55244f74784973c48de473d82e0e06b508ffac75690df7225281ea7" exitCode=0
Jul 07 15:06:56.334343 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:06:56.333885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" event={"ID":"91eec569-d4d4-406f-8927-f67a43479ae6","Type":"ContainerDied","Data":"dbd4b8a3e55244f74784973c48de473d82e0e06b508ffac75690df7225281ea7"}
Jul 07 15:07:10.413065 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:10.413028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" event={"ID":"91eec569-d4d4-406f-8927-f67a43479ae6","Type":"ContainerStarted","Data":"999a506200b0d2ef087d66c751afdfb6cd02e47ccaad7bc2ad762c5990e5f7f7"}
Jul 07 15:07:13.430267 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:13.430233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" event={"ID":"91eec569-d4d4-406f-8927-f67a43479ae6","Type":"ContainerStarted","Data":"15f9167d7420723c6f1e700d4ab65fce998485da40edaa20bb488eecd3b7e482"}
Jul 07 15:07:16.447104 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:16.447063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" event={"ID":"91eec569-d4d4-406f-8927-f67a43479ae6","Type":"ContainerStarted","Data":"a054eda9e6a4789c5ec6425b0100ebfae9ac05fe533c45a3d2a6f6b9c85394be"}
Jul 07 15:07:16.447556 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:16.447285    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:07:16.447556 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:16.447475    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:07:16.448750 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:16.448717    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 15:07:16.471430 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:16.471373    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podStartSLOduration=1.487845171 podStartE2EDuration="24.471343165s" totalImagesPullingTime="19.38255048s" totalInitContainerRuntime="3.600947514s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:06:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:07:16.468821738 +0000 UTC m=+598.201434074" watchObservedRunningTime="2026-07-07 15:07:16.471343165 +0000 UTC m=+598.203955485"
Jul 07 15:07:17.451099 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:17.451071    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:07:17.451593 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:17.451233    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 15:07:17.452338 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:17.452309    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:07:17.455235 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:17.455216    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:07:18.455925 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:18.455887    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 15:07:18.456395 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:18.456229    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:07:19.460094 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:19.460034    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 15:07:19.460608 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:19.460228    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:07:29.460648 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:29.460603    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 15:07:29.461220 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:29.461057    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:07:39.460655 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:39.460600    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 15:07:39.461078 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:39.460996    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:07:49.459989 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:49.459882    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 15:07:49.460504 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:49.460281    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:07:59.460615 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:59.460554    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 15:07:59.461131 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:07:59.461092    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:08:09.459995 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:09.459947    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 15:08:09.460512 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:09.460391    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:08:19.459935 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:19.459883    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 15:08:19.462404 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:19.460283    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:08:29.460533 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:29.460504    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:08:29.460934 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:29.460852    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:08:37.143069 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.143034    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"]
Jul 07 15:08:37.143589 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.143310    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container" containerID="cri-o://999a506200b0d2ef087d66c751afdfb6cd02e47ccaad7bc2ad762c5990e5f7f7" gracePeriod=30
Jul 07 15:08:37.143589 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.143388    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kube-rbac-proxy" containerID="cri-o://15f9167d7420723c6f1e700d4ab65fce998485da40edaa20bb488eecd3b7e482" gracePeriod=30
Jul 07 15:08:37.143797 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.143676    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent" containerID="cri-o://a054eda9e6a4789c5ec6425b0100ebfae9ac05fe533c45a3d2a6f6b9c85394be" gracePeriod=30
Jul 07 15:08:37.148385 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.148335    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"]
Jul 07 15:08:37.152659 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.152496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.156271 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.156249    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-e3a4a-kube-rbac-proxy-sar-config\""
Jul 07 15:08:37.156396 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.156305    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-e3a4a-predictor-serving-cert\""
Jul 07 15:08:37.164385 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.164335    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"]
Jul 07 15:08:37.223119 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.223089    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"]
Jul 07 15:08:37.226941 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.226918    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.229819 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.229794    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-e3a4a-kube-rbac-proxy-sar-config\""
Jul 07 15:08:37.229819 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.229812    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-e3a4a-predictor-serving-cert\""
Jul 07 15:08:37.238502 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.238477    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"]
Jul 07 15:08:37.252332 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.252304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ndzh\" (UniqueName: \"kubernetes.io/projected/8ec8ada2-35b3-483d-be07-e4fe373dd681-kube-api-access-2ndzh\") pod \"isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.252332 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.252339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ec8ada2-35b3-483d-be07-e4fe373dd681-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.252593 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.252429    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a71558c8-c93c-4383-9db1-041926890e0b-proxy-tls\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.252593 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.252471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a71558c8-c93c-4383-9db1-041926890e0b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.252593 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.252539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a71558c8-c93c-4383-9db1-041926890e0b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.252593 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.252571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ec8ada2-35b3-483d-be07-e4fe373dd681-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.252811 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.252622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ec8ada2-35b3-483d-be07-e4fe373dd681-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.252811 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.252707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nngdx\" (UniqueName: \"kubernetes.io/projected/a71558c8-c93c-4383-9db1-041926890e0b-kube-api-access-nngdx\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.252811 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.252734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a71558c8-c93c-4383-9db1-041926890e0b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.252811 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.252758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ec8ada2-35b3-483d-be07-e4fe373dd681-proxy-tls\") pod \"isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.354001 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.353956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ndzh\" (UniqueName: \"kubernetes.io/projected/8ec8ada2-35b3-483d-be07-e4fe373dd681-kube-api-access-2ndzh\") pod \"isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.354186 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.354016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ec8ada2-35b3-483d-be07-e4fe373dd681-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.354186 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.354055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a71558c8-c93c-4383-9db1-041926890e0b-proxy-tls\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.354186 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.354077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a71558c8-c93c-4383-9db1-041926890e0b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.354186 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.354102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a71558c8-c93c-4383-9db1-041926890e0b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.354186 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.354123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ec8ada2-35b3-483d-be07-e4fe373dd681-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.354186 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.354175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ec8ada2-35b3-483d-be07-e4fe373dd681-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.354543 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.354218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nngdx\" (UniqueName: \"kubernetes.io/projected/a71558c8-c93c-4383-9db1-041926890e0b-kube-api-access-nngdx\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.354543 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.354244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a71558c8-c93c-4383-9db1-041926890e0b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.354543 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.354367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ec8ada2-35b3-483d-be07-e4fe373dd681-proxy-tls\") pod \"isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.354543 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.354484    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ec8ada2-35b3-483d-be07-e4fe373dd681-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.354745 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:08:37.354617    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-serving-cert: secret "isvc-sklearn-graph-raw-e3a4a-predictor-serving-cert" not found
Jul 07 15:08:37.354745 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:08:37.354692    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a71558c8-c93c-4383-9db1-041926890e0b-proxy-tls podName:a71558c8-c93c-4383-9db1-041926890e0b nodeName:}" failed. No retries permitted until 2026-07-07 15:08:37.854670689 +0000 UTC m=+679.587282986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a71558c8-c93c-4383-9db1-041926890e0b-proxy-tls") pod "isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" (UID: "a71558c8-c93c-4383-9db1-041926890e0b") : secret "isvc-sklearn-graph-raw-e3a4a-predictor-serving-cert" not found
Jul 07 15:08:37.354745 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.354712    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a71558c8-c93c-4383-9db1-041926890e0b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.354976 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.354955    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a71558c8-c93c-4383-9db1-041926890e0b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.355043 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.355013    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a71558c8-c93c-4383-9db1-041926890e0b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.355267 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.355247    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ec8ada2-35b3-483d-be07-e4fe373dd681-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.355388 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.355365    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ec8ada2-35b3-483d-be07-e4fe373dd681-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.357062 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.357041    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ec8ada2-35b3-483d-be07-e4fe373dd681-proxy-tls\") pod \"isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.365657 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.365636    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ndzh\" (UniqueName: \"kubernetes.io/projected/8ec8ada2-35b3-483d-be07-e4fe373dd681-kube-api-access-2ndzh\") pod \"isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.370464 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.370444    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nngdx\" (UniqueName: \"kubernetes.io/projected/a71558c8-c93c-4383-9db1-041926890e0b-kube-api-access-nngdx\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.451988 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.451942    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 07 15:08:37.537617 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.537584    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:08:37.680647 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.680612    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"]
Jul 07 15:08:37.683730 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:08:37.683691    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec8ada2_35b3_483d_be07_e4fe373dd681.slice/crio-5da2731eba0f3931c0f1c26087d47ca42dfa9ab1f10832a99f657a019430ac09 WatchSource:0}: Error finding container 5da2731eba0f3931c0f1c26087d47ca42dfa9ab1f10832a99f657a019430ac09: Status 404 returned error can't find the container with id 5da2731eba0f3931c0f1c26087d47ca42dfa9ab1f10832a99f657a019430ac09
Jul 07 15:08:37.801877 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.801845    2581 generic.go:356] "Generic (PLEG): container finished" podID="91eec569-d4d4-406f-8927-f67a43479ae6" containerID="15f9167d7420723c6f1e700d4ab65fce998485da40edaa20bb488eecd3b7e482" exitCode=2
Jul 07 15:08:37.802048 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.801921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" event={"ID":"91eec569-d4d4-406f-8927-f67a43479ae6","Type":"ContainerDied","Data":"15f9167d7420723c6f1e700d4ab65fce998485da40edaa20bb488eecd3b7e482"}
Jul 07 15:08:37.803475 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.803453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" event={"ID":"8ec8ada2-35b3-483d-be07-e4fe373dd681","Type":"ContainerStarted","Data":"a69c03c91c5871687f5c3c682363ea79496d33b94feeb23739ddc75c5c7e359e"}
Jul 07 15:08:37.803584 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.803496    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" event={"ID":"8ec8ada2-35b3-483d-be07-e4fe373dd681","Type":"ContainerStarted","Data":"5da2731eba0f3931c0f1c26087d47ca42dfa9ab1f10832a99f657a019430ac09"}
Jul 07 15:08:37.859682 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.859646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a71558c8-c93c-4383-9db1-041926890e0b-proxy-tls\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:37.862014 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:37.861992    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a71558c8-c93c-4383-9db1-041926890e0b-proxy-tls\") pod \"isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:38.072837 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:38.072739    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:38.233669 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:38.233630    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"]
Jul 07 15:08:38.235913 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:08:38.235889    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71558c8_c93c_4383_9db1_041926890e0b.slice/crio-879ead6184961b2a08613ad6b8491ca8b9b59b168a536358bf504850870b1ac4 WatchSource:0}: Error finding container 879ead6184961b2a08613ad6b8491ca8b9b59b168a536358bf504850870b1ac4: Status 404 returned error can't find the container with id 879ead6184961b2a08613ad6b8491ca8b9b59b168a536358bf504850870b1ac4
Jul 07 15:08:38.810413 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:38.810367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" event={"ID":"a71558c8-c93c-4383-9db1-041926890e0b","Type":"ContainerStarted","Data":"324e250de90ecf143ba6e6a9c3a1fd6f8b8975d2d6b9536019c58b4976cb25b8"}
Jul 07 15:08:38.810568 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:38.810455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" event={"ID":"a71558c8-c93c-4383-9db1-041926890e0b","Type":"ContainerStarted","Data":"879ead6184961b2a08613ad6b8491ca8b9b59b168a536358bf504850870b1ac4"}
Jul 07 15:08:39.460456 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:39.460408    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 15:08:39.460841 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:39.460738    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:08:41.826457 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:41.826420    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerID="a69c03c91c5871687f5c3c682363ea79496d33b94feeb23739ddc75c5c7e359e" exitCode=0
Jul 07 15:08:41.826961 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:41.826502    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" event={"ID":"8ec8ada2-35b3-483d-be07-e4fe373dd681","Type":"ContainerDied","Data":"a69c03c91c5871687f5c3c682363ea79496d33b94feeb23739ddc75c5c7e359e"}
Jul 07 15:08:42.451865 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:42.451810    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 07 15:08:42.835447 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:42.835177    2581 generic.go:356] "Generic (PLEG): container finished" podID="a71558c8-c93c-4383-9db1-041926890e0b" containerID="324e250de90ecf143ba6e6a9c3a1fd6f8b8975d2d6b9536019c58b4976cb25b8" exitCode=0
Jul 07 15:08:42.835447 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:42.835208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" event={"ID":"a71558c8-c93c-4383-9db1-041926890e0b","Type":"ContainerDied","Data":"324e250de90ecf143ba6e6a9c3a1fd6f8b8975d2d6b9536019c58b4976cb25b8"}
Jul 07 15:08:42.840208 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:42.840180    2581 generic.go:356] "Generic (PLEG): container finished" podID="91eec569-d4d4-406f-8927-f67a43479ae6" containerID="999a506200b0d2ef087d66c751afdfb6cd02e47ccaad7bc2ad762c5990e5f7f7" exitCode=0
Jul 07 15:08:42.840314 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:42.840222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" event={"ID":"91eec569-d4d4-406f-8927-f67a43479ae6","Type":"ContainerDied","Data":"999a506200b0d2ef087d66c751afdfb6cd02e47ccaad7bc2ad762c5990e5f7f7"}
Jul 07 15:08:43.849926 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:43.849884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" event={"ID":"a71558c8-c93c-4383-9db1-041926890e0b","Type":"ContainerStarted","Data":"5dba4b1382a4a9b40fd98129d0b0a11d9d6164904d085aaa1c90719b59893323"}
Jul 07 15:08:43.850406 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:43.849987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" event={"ID":"a71558c8-c93c-4383-9db1-041926890e0b","Type":"ContainerStarted","Data":"9925fbec845ad7a0f7f774480ff630b3e79378e01b629133b66b51c93650f33c"}
Jul 07 15:08:43.850483 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:43.850466    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:43.850761 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:43.850736    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:43.852007 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:43.851970    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:08:43.871177 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:43.871116    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podStartSLOduration=3.293969072 podStartE2EDuration="6.871097143s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.577128071s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:08:43.869984567 +0000 UTC m=+685.602596902" watchObservedRunningTime="2026-07-07 15:08:43.871097143 +0000 UTC m=+685.603709464"
Jul 07 15:08:44.855447 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:44.855401    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:08:45.863878 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:45.863839    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:08:47.452096 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:47.452046    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 07 15:08:47.452588 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:47.452210    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:08:49.459978 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:49.459887    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 15:08:49.460600 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:49.460565    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:08:50.920540 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:50.920498    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:08:50.921378 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:50.921279    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:08:52.452226 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:52.452132    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 07 15:08:57.452094 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:57.452040    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 07 15:08:59.460388 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:59.460323    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 15:08:59.460865 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:59.460563    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:08:59.460865 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:59.460741    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:08:59.460985 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:08:59.460865    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:09:00.922215 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:00.922171    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:09:02.452067 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:02.452023    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 07 15:09:03.957430 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:03.957388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" event={"ID":"8ec8ada2-35b3-483d-be07-e4fe373dd681","Type":"ContainerStarted","Data":"8c122163be890b1ed87a669a58b639f61a7053b12c13a92e100fedfe3832b64a"}
Jul 07 15:09:03.957838 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:03.957481    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" event={"ID":"8ec8ada2-35b3-483d-be07-e4fe373dd681","Type":"ContainerStarted","Data":"ff6f220af6403a27a6a379ecea186d7562edb2581ebb2c5da5ee7cd8cee574bd"}
Jul 07 15:09:03.957838 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:03.957743    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:09:03.957945 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:03.957891    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:09:03.959076 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:03.959050    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 15:09:03.978570 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:03.978520    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" podStartSLOduration=1.850965023 podStartE2EDuration="26.97850493s" totalImagesPullingTime="21.162187431s" totalInitContainerRuntime="3.965352476s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:09:03.977482657 +0000 UTC m=+705.710094979" watchObservedRunningTime="2026-07-07 15:09:03.97850493 +0000 UTC m=+705.711117249"
Jul 07 15:09:04.962267 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:04.962231    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 15:09:07.298191 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.298166    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:09:07.363036 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.362999    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91eec569-d4d4-406f-8927-f67a43479ae6-cabundle-cert\") pod \"91eec569-d4d4-406f-8927-f67a43479ae6\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") "
Jul 07 15:09:07.363036 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.363034    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91eec569-d4d4-406f-8927-f67a43479ae6-kube-rbac-proxy-sar-config\") pod \"91eec569-d4d4-406f-8927-f67a43479ae6\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") "
Jul 07 15:09:07.363280 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.363073    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eec569-d4d4-406f-8927-f67a43479ae6-proxy-tls\") pod \"91eec569-d4d4-406f-8927-f67a43479ae6\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") "
Jul 07 15:09:07.363280 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.363099    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6ss9\" (UniqueName: \"kubernetes.io/projected/91eec569-d4d4-406f-8927-f67a43479ae6-kube-api-access-h6ss9\") pod \"91eec569-d4d4-406f-8927-f67a43479ae6\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") "
Jul 07 15:09:07.363280 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.363208    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91eec569-d4d4-406f-8927-f67a43479ae6-kserve-provision-location\") pod \"91eec569-d4d4-406f-8927-f67a43479ae6\" (UID: \"91eec569-d4d4-406f-8927-f67a43479ae6\") "
Jul 07 15:09:07.363474 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.363456    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91eec569-d4d4-406f-8927-f67a43479ae6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "91eec569-d4d4-406f-8927-f67a43479ae6" (UID: "91eec569-d4d4-406f-8927-f67a43479ae6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:09:07.363544 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.363513    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91eec569-d4d4-406f-8927-f67a43479ae6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "91eec569-d4d4-406f-8927-f67a43479ae6" (UID: "91eec569-d4d4-406f-8927-f67a43479ae6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:09:07.363596 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.363567    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91eec569-d4d4-406f-8927-f67a43479ae6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "91eec569-d4d4-406f-8927-f67a43479ae6" (UID: "91eec569-d4d4-406f-8927-f67a43479ae6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:09:07.365291 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.365268    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91eec569-d4d4-406f-8927-f67a43479ae6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91eec569-d4d4-406f-8927-f67a43479ae6" (UID: "91eec569-d4d4-406f-8927-f67a43479ae6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:09:07.365712 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.365693    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91eec569-d4d4-406f-8927-f67a43479ae6-kube-api-access-h6ss9" (OuterVolumeSpecName: "kube-api-access-h6ss9") pod "91eec569-d4d4-406f-8927-f67a43479ae6" (UID: "91eec569-d4d4-406f-8927-f67a43479ae6"). InnerVolumeSpecName "kube-api-access-h6ss9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:09:07.464628 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.464593    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91eec569-d4d4-406f-8927-f67a43479ae6-kserve-provision-location\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:09:07.464628 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.464622    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91eec569-d4d4-406f-8927-f67a43479ae6-cabundle-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:09:07.464628 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.464632    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91eec569-d4d4-406f-8927-f67a43479ae6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:09:07.464869 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.464642    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eec569-d4d4-406f-8927-f67a43479ae6-proxy-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:09:07.464869 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.464652    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6ss9\" (UniqueName: \"kubernetes.io/projected/91eec569-d4d4-406f-8927-f67a43479ae6-kube-api-access-h6ss9\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:09:07.980871 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.980825    2581 generic.go:356] "Generic (PLEG): container finished" podID="91eec569-d4d4-406f-8927-f67a43479ae6" containerID="a054eda9e6a4789c5ec6425b0100ebfae9ac05fe533c45a3d2a6f6b9c85394be" exitCode=0
Jul 07 15:09:07.980871 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.980876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" event={"ID":"91eec569-d4d4-406f-8927-f67a43479ae6","Type":"ContainerDied","Data":"a054eda9e6a4789c5ec6425b0100ebfae9ac05fe533c45a3d2a6f6b9c85394be"}
Jul 07 15:09:07.981127 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.980908    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"
Jul 07 15:09:07.981127 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.980933    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9" event={"ID":"91eec569-d4d4-406f-8927-f67a43479ae6","Type":"ContainerDied","Data":"d96ac4f2ba6e7455b2d696dd0254c73db05464715e2b044703e686b159f843b7"}
Jul 07 15:09:07.981127 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.980974    2581 scope.go:117] "RemoveContainer" containerID="a054eda9e6a4789c5ec6425b0100ebfae9ac05fe533c45a3d2a6f6b9c85394be"
Jul 07 15:09:07.990030 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.990011    2581 scope.go:117] "RemoveContainer" containerID="15f9167d7420723c6f1e700d4ab65fce998485da40edaa20bb488eecd3b7e482"
Jul 07 15:09:07.998586 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:07.998569    2581 scope.go:117] "RemoveContainer" containerID="999a506200b0d2ef087d66c751afdfb6cd02e47ccaad7bc2ad762c5990e5f7f7"
Jul 07 15:09:08.004670 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:08.004644    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"]
Jul 07 15:09:08.006707 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:08.006622    2581 scope.go:117] "RemoveContainer" containerID="dbd4b8a3e55244f74784973c48de473d82e0e06b508ffac75690df7225281ea7"
Jul 07 15:09:08.009013 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:08.008995    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-df08e-predictor-6c5fdf897c-jjtx9"]
Jul 07 15:09:08.014317 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:08.014289    2581 scope.go:117] "RemoveContainer" containerID="a054eda9e6a4789c5ec6425b0100ebfae9ac05fe533c45a3d2a6f6b9c85394be"
Jul 07 15:09:08.014627 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:09:08.014604    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a054eda9e6a4789c5ec6425b0100ebfae9ac05fe533c45a3d2a6f6b9c85394be\": container with ID starting with a054eda9e6a4789c5ec6425b0100ebfae9ac05fe533c45a3d2a6f6b9c85394be not found: ID does not exist" containerID="a054eda9e6a4789c5ec6425b0100ebfae9ac05fe533c45a3d2a6f6b9c85394be"
Jul 07 15:09:08.014691 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:08.014639    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a054eda9e6a4789c5ec6425b0100ebfae9ac05fe533c45a3d2a6f6b9c85394be"} err="failed to get container status \"a054eda9e6a4789c5ec6425b0100ebfae9ac05fe533c45a3d2a6f6b9c85394be\": rpc error: code = NotFound desc = could not find container \"a054eda9e6a4789c5ec6425b0100ebfae9ac05fe533c45a3d2a6f6b9c85394be\": container with ID starting with a054eda9e6a4789c5ec6425b0100ebfae9ac05fe533c45a3d2a6f6b9c85394be not found: ID does not exist"
Jul 07 15:09:08.014691 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:08.014676    2581 scope.go:117] "RemoveContainer" containerID="15f9167d7420723c6f1e700d4ab65fce998485da40edaa20bb488eecd3b7e482"
Jul 07 15:09:08.014941 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:09:08.014921    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f9167d7420723c6f1e700d4ab65fce998485da40edaa20bb488eecd3b7e482\": container with ID starting with 15f9167d7420723c6f1e700d4ab65fce998485da40edaa20bb488eecd3b7e482 not found: ID does not exist" containerID="15f9167d7420723c6f1e700d4ab65fce998485da40edaa20bb488eecd3b7e482"
Jul 07 15:09:08.014982 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:08.014947    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f9167d7420723c6f1e700d4ab65fce998485da40edaa20bb488eecd3b7e482"} err="failed to get container status \"15f9167d7420723c6f1e700d4ab65fce998485da40edaa20bb488eecd3b7e482\": rpc error: code = NotFound desc = could not find container \"15f9167d7420723c6f1e700d4ab65fce998485da40edaa20bb488eecd3b7e482\": container with ID starting with 15f9167d7420723c6f1e700d4ab65fce998485da40edaa20bb488eecd3b7e482 not found: ID does not exist"
Jul 07 15:09:08.014982 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:08.014965    2581 scope.go:117] "RemoveContainer" containerID="999a506200b0d2ef087d66c751afdfb6cd02e47ccaad7bc2ad762c5990e5f7f7"
Jul 07 15:09:08.015182 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:09:08.015166    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999a506200b0d2ef087d66c751afdfb6cd02e47ccaad7bc2ad762c5990e5f7f7\": container with ID starting with 999a506200b0d2ef087d66c751afdfb6cd02e47ccaad7bc2ad762c5990e5f7f7 not found: ID does not exist" containerID="999a506200b0d2ef087d66c751afdfb6cd02e47ccaad7bc2ad762c5990e5f7f7"
Jul 07 15:09:08.015217 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:08.015185    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999a506200b0d2ef087d66c751afdfb6cd02e47ccaad7bc2ad762c5990e5f7f7"} err="failed to get container status \"999a506200b0d2ef087d66c751afdfb6cd02e47ccaad7bc2ad762c5990e5f7f7\": rpc error: code = NotFound desc = could not find container \"999a506200b0d2ef087d66c751afdfb6cd02e47ccaad7bc2ad762c5990e5f7f7\": container with ID starting with 999a506200b0d2ef087d66c751afdfb6cd02e47ccaad7bc2ad762c5990e5f7f7 not found: ID does not exist"
Jul 07 15:09:08.015217 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:08.015198    2581 scope.go:117] "RemoveContainer" containerID="dbd4b8a3e55244f74784973c48de473d82e0e06b508ffac75690df7225281ea7"
Jul 07 15:09:08.015425 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:09:08.015407    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd4b8a3e55244f74784973c48de473d82e0e06b508ffac75690df7225281ea7\": container with ID starting with dbd4b8a3e55244f74784973c48de473d82e0e06b508ffac75690df7225281ea7 not found: ID does not exist" containerID="dbd4b8a3e55244f74784973c48de473d82e0e06b508ffac75690df7225281ea7"
Jul 07 15:09:08.015472 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:08.015430    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd4b8a3e55244f74784973c48de473d82e0e06b508ffac75690df7225281ea7"} err="failed to get container status \"dbd4b8a3e55244f74784973c48de473d82e0e06b508ffac75690df7225281ea7\": rpc error: code = NotFound desc = could not find container \"dbd4b8a3e55244f74784973c48de473d82e0e06b508ffac75690df7225281ea7\": container with ID starting with dbd4b8a3e55244f74784973c48de473d82e0e06b508ffac75690df7225281ea7 not found: ID does not exist"
Jul 07 15:09:08.938734 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:08.938699    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" path="/var/lib/kubelet/pods/91eec569-d4d4-406f-8927-f67a43479ae6/volumes"
Jul 07 15:09:09.967006 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:09.966980    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:09:09.967573 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:09.967545    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 15:09:10.921976 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:10.921920    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:09:19.968417 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:19.968378    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 15:09:20.922191 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:20.922131    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:09:29.967766 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:29.967725    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 15:09:30.921388 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:30.921319    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:09:39.967474 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:39.967433    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 15:09:40.921441 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:40.921396    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:09:49.968091 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:49.968054    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 15:09:50.921411 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:50.921341    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:09:59.967976 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:09:59.967931    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 15:10:00.921713 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:00.921659    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:10:09.969058 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:09.969022    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:10:10.922086 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:10.922049    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:10:37.455692 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.455651    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"]
Jul 07 15:10:37.456298 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.455971    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container" containerID="cri-o://9925fbec845ad7a0f7f774480ff630b3e79378e01b629133b66b51c93650f33c" gracePeriod=30
Jul 07 15:10:37.456298 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.456228    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kube-rbac-proxy" containerID="cri-o://5dba4b1382a4a9b40fd98129d0b0a11d9d6164904d085aaa1c90719b59893323" gracePeriod=30
Jul 07 15:10:37.460041 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.460009    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"]
Jul 07 15:10:37.460553 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.460533    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="storage-initializer"
Jul 07 15:10:37.460553 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.460552    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="storage-initializer"
Jul 07 15:10:37.460671 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.460568    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container"
Jul 07 15:10:37.460671 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.460576    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container"
Jul 07 15:10:37.460671 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.460591    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kube-rbac-proxy"
Jul 07 15:10:37.460671 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.460599    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kube-rbac-proxy"
Jul 07 15:10:37.460671 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.460615    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent"
Jul 07 15:10:37.460671 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.460623    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent"
Jul 07 15:10:37.460846 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.460693    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kserve-container"
Jul 07 15:10:37.460846 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.460705    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="agent"
Jul 07 15:10:37.460846 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.460714    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="91eec569-d4d4-406f-8927-f67a43479ae6" containerName="kube-rbac-proxy"
Jul 07 15:10:37.463546 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.463529    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.466147 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.466128    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-serving-cert\""
Jul 07 15:10:37.466255 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.466242    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-c0f7d-kube-rbac-proxy-sar-config\""
Jul 07 15:10:37.489821 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.489794    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"]
Jul 07 15:10:37.544882 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.544848    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"]
Jul 07 15:10:37.548212 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.548179    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:37.552407 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.552379    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-serving-cert\""
Jul 07 15:10:37.552557 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.552392    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-c0f7d-kube-rbac-proxy-sar-config\""
Jul 07 15:10:37.568793 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.568747    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"]
Jul 07 15:10:37.601789 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.601748    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"]
Jul 07 15:10:37.602038 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.601995    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kserve-container" containerID="cri-o://ff6f220af6403a27a6a379ecea186d7562edb2581ebb2c5da5ee7cd8cee574bd" gracePeriod=30
Jul 07 15:10:37.602119 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.602021    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kube-rbac-proxy" containerID="cri-o://8c122163be890b1ed87a669a58b639f61a7053b12c13a92e100fedfe3832b64a" gracePeriod=30
Jul 07 15:10:37.658394 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.658344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cd44159-004a-40b3-9a1e-90596e648f90-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.658552 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.658425    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26714cf3-2b63-4b65-90c4-6b71822b067d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:37.658552 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.658468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dskl5\" (UniqueName: \"kubernetes.io/projected/26714cf3-2b63-4b65-90c4-6b71822b067d-kube-api-access-dskl5\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:37.658552 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.658489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26714cf3-2b63-4b65-90c4-6b71822b067d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:37.658552 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.658517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26714cf3-2b63-4b65-90c4-6b71822b067d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:37.658690 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.658563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd44159-004a-40b3-9a1e-90596e648f90-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.658690 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.658596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cd44159-004a-40b3-9a1e-90596e648f90-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.658690 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.658623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26714cf3-2b63-4b65-90c4-6b71822b067d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:37.658690 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.658667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd9bg\" (UniqueName: \"kubernetes.io/projected/3cd44159-004a-40b3-9a1e-90596e648f90-kube-api-access-dd9bg\") pod \"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.658806 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.658740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cd44159-004a-40b3-9a1e-90596e648f90-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.759593 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.759555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cd44159-004a-40b3-9a1e-90596e648f90-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.759775 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.759604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26714cf3-2b63-4b65-90c4-6b71822b067d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:37.759775 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.759653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd9bg\" (UniqueName: \"kubernetes.io/projected/3cd44159-004a-40b3-9a1e-90596e648f90-kube-api-access-dd9bg\") pod \"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.759892 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.759858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cd44159-004a-40b3-9a1e-90596e648f90-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.759957 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.759932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cd44159-004a-40b3-9a1e-90596e648f90-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.760000 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.759970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26714cf3-2b63-4b65-90c4-6b71822b067d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:37.760000 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.759995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dskl5\" (UniqueName: \"kubernetes.io/projected/26714cf3-2b63-4b65-90c4-6b71822b067d-kube-api-access-dskl5\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:37.760108 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.760027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26714cf3-2b63-4b65-90c4-6b71822b067d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:37.760108 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.760093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26714cf3-2b63-4b65-90c4-6b71822b067d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:37.760276 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.760122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd44159-004a-40b3-9a1e-90596e648f90-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.760276 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:10:37.760183    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-c0f7d-predictor-serving-cert" not found
Jul 07 15:10:37.760276 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:10:37.760264    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26714cf3-2b63-4b65-90c4-6b71822b067d-proxy-tls podName:26714cf3-2b63-4b65-90c4-6b71822b067d nodeName:}" failed. No retries permitted until 2026-07-07 15:10:38.260242658 +0000 UTC m=+799.992854965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/26714cf3-2b63-4b65-90c4-6b71822b067d-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" (UID: "26714cf3-2b63-4b65-90c4-6b71822b067d") : secret "isvc-xgboost-graph-raw-hpa-c0f7d-predictor-serving-cert" not found
Jul 07 15:10:37.760475 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.760460    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26714cf3-2b63-4b65-90c4-6b71822b067d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:37.760600 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.760569    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd44159-004a-40b3-9a1e-90596e648f90-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.760693 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.760664    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cd44159-004a-40b3-9a1e-90596e648f90-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.760774 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.760668    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26714cf3-2b63-4b65-90c4-6b71822b067d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:37.760774 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.760718    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cd44159-004a-40b3-9a1e-90596e648f90-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.761015 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.760998    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26714cf3-2b63-4b65-90c4-6b71822b067d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:37.762751 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.762726    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cd44159-004a-40b3-9a1e-90596e648f90-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.770131 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.770107    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd9bg\" (UniqueName: \"kubernetes.io/projected/3cd44159-004a-40b3-9a1e-90596e648f90-kube-api-access-dd9bg\") pod \"isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.771191 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.771172    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dskl5\" (UniqueName: \"kubernetes.io/projected/26714cf3-2b63-4b65-90c4-6b71822b067d-kube-api-access-dskl5\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:37.780045 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.780024    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:37.922375 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:37.922328    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"]
Jul 07 15:10:37.924899 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:10:37.924872    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd44159_004a_40b3_9a1e_90596e648f90.slice/crio-a74bde79593d0e240f1dab3848220537435146e6bee3c63b408234f7666279b7 WatchSource:0}: Error finding container a74bde79593d0e240f1dab3848220537435146e6bee3c63b408234f7666279b7: Status 404 returned error can't find the container with id a74bde79593d0e240f1dab3848220537435146e6bee3c63b408234f7666279b7
Jul 07 15:10:38.265875 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:38.265787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26714cf3-2b63-4b65-90c4-6b71822b067d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:38.268111 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:38.268087    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26714cf3-2b63-4b65-90c4-6b71822b067d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:38.376612 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:38.376580    2581 generic.go:356] "Generic (PLEG): container finished" podID="a71558c8-c93c-4383-9db1-041926890e0b" containerID="5dba4b1382a4a9b40fd98129d0b0a11d9d6164904d085aaa1c90719b59893323" exitCode=2
Jul 07 15:10:38.376785 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:38.376655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" event={"ID":"a71558c8-c93c-4383-9db1-041926890e0b","Type":"ContainerDied","Data":"5dba4b1382a4a9b40fd98129d0b0a11d9d6164904d085aaa1c90719b59893323"}
Jul 07 15:10:38.378748 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:38.378715    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerID="8c122163be890b1ed87a669a58b639f61a7053b12c13a92e100fedfe3832b64a" exitCode=2
Jul 07 15:10:38.378892 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:38.378786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" event={"ID":"8ec8ada2-35b3-483d-be07-e4fe373dd681","Type":"ContainerDied","Data":"8c122163be890b1ed87a669a58b639f61a7053b12c13a92e100fedfe3832b64a"}
Jul 07 15:10:38.380262 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:38.380229    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" event={"ID":"3cd44159-004a-40b3-9a1e-90596e648f90","Type":"ContainerStarted","Data":"d35e89bbd1abfbba934448cbb56af17af0e8d55a8758dad9b0c67940134b35ac"}
Jul 07 15:10:38.380394 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:38.380283    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" event={"ID":"3cd44159-004a-40b3-9a1e-90596e648f90","Type":"ContainerStarted","Data":"a74bde79593d0e240f1dab3848220537435146e6bee3c63b408234f7666279b7"}
Jul 07 15:10:38.463758 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:38.463726    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:38.601674 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:38.601651    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"]
Jul 07 15:10:38.603765 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:10:38.603735    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26714cf3_2b63_4b65_90c4_6b71822b067d.slice/crio-afd7d873a97dba74eb58363770b77ee1a5cb967bac31e0530573db7401076d1f WatchSource:0}: Error finding container afd7d873a97dba74eb58363770b77ee1a5cb967bac31e0530573db7401076d1f: Status 404 returned error can't find the container with id afd7d873a97dba74eb58363770b77ee1a5cb967bac31e0530573db7401076d1f
Jul 07 15:10:39.386721 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:39.386681    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" event={"ID":"26714cf3-2b63-4b65-90c4-6b71822b067d","Type":"ContainerStarted","Data":"f67f7cc6bfdd9c665dde3eea015110cb7a478cc8003f92261983c3efb9114911"}
Jul 07 15:10:39.386919 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:39.386748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" event={"ID":"26714cf3-2b63-4b65-90c4-6b71822b067d","Type":"ContainerStarted","Data":"afd7d873a97dba74eb58363770b77ee1a5cb967bac31e0530573db7401076d1f"}
Jul 07 15:10:39.962552 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:39.962506    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 07 15:10:39.967463 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:39.967429    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 15:10:40.864979 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:40.864928    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 07 15:10:40.922224 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:40.922180    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:10:41.342651 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.342626    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:10:41.394300 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.394220    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ec8ada2-35b3-483d-be07-e4fe373dd681-kserve-provision-location\") pod \"8ec8ada2-35b3-483d-be07-e4fe373dd681\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") "
Jul 07 15:10:41.394300 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.394260    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ec8ada2-35b3-483d-be07-e4fe373dd681-cabundle-cert\") pod \"8ec8ada2-35b3-483d-be07-e4fe373dd681\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") "
Jul 07 15:10:41.394300 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.394284    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ec8ada2-35b3-483d-be07-e4fe373dd681-kube-rbac-proxy-sar-config\") pod \"8ec8ada2-35b3-483d-be07-e4fe373dd681\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") "
Jul 07 15:10:41.394597 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.394324    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ndzh\" (UniqueName: \"kubernetes.io/projected/8ec8ada2-35b3-483d-be07-e4fe373dd681-kube-api-access-2ndzh\") pod \"8ec8ada2-35b3-483d-be07-e4fe373dd681\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") "
Jul 07 15:10:41.394597 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.394428    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ec8ada2-35b3-483d-be07-e4fe373dd681-proxy-tls\") pod \"8ec8ada2-35b3-483d-be07-e4fe373dd681\" (UID: \"8ec8ada2-35b3-483d-be07-e4fe373dd681\") "
Jul 07 15:10:41.394707 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.394626    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec8ada2-35b3-483d-be07-e4fe373dd681-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ec8ada2-35b3-483d-be07-e4fe373dd681" (UID: "8ec8ada2-35b3-483d-be07-e4fe373dd681"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:10:41.394707 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.394649    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec8ada2-35b3-483d-be07-e4fe373dd681-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ec8ada2-35b3-483d-be07-e4fe373dd681" (UID: "8ec8ada2-35b3-483d-be07-e4fe373dd681"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:10:41.394707 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.394680    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec8ada2-35b3-483d-be07-e4fe373dd681-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ec8ada2-35b3-483d-be07-e4fe373dd681" (UID: "8ec8ada2-35b3-483d-be07-e4fe373dd681"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:10:41.396520 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.396491    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec8ada2-35b3-483d-be07-e4fe373dd681-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ec8ada2-35b3-483d-be07-e4fe373dd681" (UID: "8ec8ada2-35b3-483d-be07-e4fe373dd681"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:10:41.396624 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.396551    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec8ada2-35b3-483d-be07-e4fe373dd681-kube-api-access-2ndzh" (OuterVolumeSpecName: "kube-api-access-2ndzh") pod "8ec8ada2-35b3-483d-be07-e4fe373dd681" (UID: "8ec8ada2-35b3-483d-be07-e4fe373dd681"). InnerVolumeSpecName "kube-api-access-2ndzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:10:41.404001 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.403975    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerID="ff6f220af6403a27a6a379ecea186d7562edb2581ebb2c5da5ee7cd8cee574bd" exitCode=0
Jul 07 15:10:41.404081 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.404031    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"
Jul 07 15:10:41.404081 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.404054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" event={"ID":"8ec8ada2-35b3-483d-be07-e4fe373dd681","Type":"ContainerDied","Data":"ff6f220af6403a27a6a379ecea186d7562edb2581ebb2c5da5ee7cd8cee574bd"}
Jul 07 15:10:41.404155 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.404129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh" event={"ID":"8ec8ada2-35b3-483d-be07-e4fe373dd681","Type":"ContainerDied","Data":"5da2731eba0f3931c0f1c26087d47ca42dfa9ab1f10832a99f657a019430ac09"}
Jul 07 15:10:41.404189 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.404170    2581 scope.go:117] "RemoveContainer" containerID="8c122163be890b1ed87a669a58b639f61a7053b12c13a92e100fedfe3832b64a"
Jul 07 15:10:41.413937 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.413920    2581 scope.go:117] "RemoveContainer" containerID="ff6f220af6403a27a6a379ecea186d7562edb2581ebb2c5da5ee7cd8cee574bd"
Jul 07 15:10:41.421430 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.421411    2581 scope.go:117] "RemoveContainer" containerID="a69c03c91c5871687f5c3c682363ea79496d33b94feeb23739ddc75c5c7e359e"
Jul 07 15:10:41.426669 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.426646    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"]
Jul 07 15:10:41.429250 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.429231    2581 scope.go:117] "RemoveContainer" containerID="8c122163be890b1ed87a669a58b639f61a7053b12c13a92e100fedfe3832b64a"
Jul 07 15:10:41.429636 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:10:41.429605    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c122163be890b1ed87a669a58b639f61a7053b12c13a92e100fedfe3832b64a\": container with ID starting with 8c122163be890b1ed87a669a58b639f61a7053b12c13a92e100fedfe3832b64a not found: ID does not exist" containerID="8c122163be890b1ed87a669a58b639f61a7053b12c13a92e100fedfe3832b64a"
Jul 07 15:10:41.429718 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.429633    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c122163be890b1ed87a669a58b639f61a7053b12c13a92e100fedfe3832b64a"} err="failed to get container status \"8c122163be890b1ed87a669a58b639f61a7053b12c13a92e100fedfe3832b64a\": rpc error: code = NotFound desc = could not find container \"8c122163be890b1ed87a669a58b639f61a7053b12c13a92e100fedfe3832b64a\": container with ID starting with 8c122163be890b1ed87a669a58b639f61a7053b12c13a92e100fedfe3832b64a not found: ID does not exist"
Jul 07 15:10:41.429718 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.429653    2581 scope.go:117] "RemoveContainer" containerID="ff6f220af6403a27a6a379ecea186d7562edb2581ebb2c5da5ee7cd8cee574bd"
Jul 07 15:10:41.429967 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:10:41.429946    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff6f220af6403a27a6a379ecea186d7562edb2581ebb2c5da5ee7cd8cee574bd\": container with ID starting with ff6f220af6403a27a6a379ecea186d7562edb2581ebb2c5da5ee7cd8cee574bd not found: ID does not exist" containerID="ff6f220af6403a27a6a379ecea186d7562edb2581ebb2c5da5ee7cd8cee574bd"
Jul 07 15:10:41.430033 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.429979    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff6f220af6403a27a6a379ecea186d7562edb2581ebb2c5da5ee7cd8cee574bd"} err="failed to get container status \"ff6f220af6403a27a6a379ecea186d7562edb2581ebb2c5da5ee7cd8cee574bd\": rpc error: code = NotFound desc = could not find container \"ff6f220af6403a27a6a379ecea186d7562edb2581ebb2c5da5ee7cd8cee574bd\": container with ID starting with ff6f220af6403a27a6a379ecea186d7562edb2581ebb2c5da5ee7cd8cee574bd not found: ID does not exist"
Jul 07 15:10:41.430033 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.430003    2581 scope.go:117] "RemoveContainer" containerID="a69c03c91c5871687f5c3c682363ea79496d33b94feeb23739ddc75c5c7e359e"
Jul 07 15:10:41.430402 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:10:41.430373    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69c03c91c5871687f5c3c682363ea79496d33b94feeb23739ddc75c5c7e359e\": container with ID starting with a69c03c91c5871687f5c3c682363ea79496d33b94feeb23739ddc75c5c7e359e not found: ID does not exist" containerID="a69c03c91c5871687f5c3c682363ea79496d33b94feeb23739ddc75c5c7e359e"
Jul 07 15:10:41.430520 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.430436    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69c03c91c5871687f5c3c682363ea79496d33b94feeb23739ddc75c5c7e359e"} err="failed to get container status \"a69c03c91c5871687f5c3c682363ea79496d33b94feeb23739ddc75c5c7e359e\": rpc error: code = NotFound desc = could not find container \"a69c03c91c5871687f5c3c682363ea79496d33b94feeb23739ddc75c5c7e359e\": container with ID starting with a69c03c91c5871687f5c3c682363ea79496d33b94feeb23739ddc75c5c7e359e not found: ID does not exist"
Jul 07 15:10:41.432412 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.432393    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3a4a-predictor-6d9989989-k98sh"]
Jul 07 15:10:41.495560 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.495532    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ec8ada2-35b3-483d-be07-e4fe373dd681-proxy-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:10:41.495560 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.495560    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ec8ada2-35b3-483d-be07-e4fe373dd681-kserve-provision-location\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:10:41.495725 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.495574    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ec8ada2-35b3-483d-be07-e4fe373dd681-cabundle-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:10:41.495725 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.495585    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ec8ada2-35b3-483d-be07-e4fe373dd681-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:10:41.495725 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:41.495594    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ndzh\" (UniqueName: \"kubernetes.io/projected/8ec8ada2-35b3-483d-be07-e4fe373dd681-kube-api-access-2ndzh\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:10:42.316182 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.316155    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:10:42.404331 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.404241    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nngdx\" (UniqueName: \"kubernetes.io/projected/a71558c8-c93c-4383-9db1-041926890e0b-kube-api-access-nngdx\") pod \"a71558c8-c93c-4383-9db1-041926890e0b\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") "
Jul 07 15:10:42.404331 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.404273    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a71558c8-c93c-4383-9db1-041926890e0b-kserve-provision-location\") pod \"a71558c8-c93c-4383-9db1-041926890e0b\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") "
Jul 07 15:10:42.404331 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.404311    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a71558c8-c93c-4383-9db1-041926890e0b-proxy-tls\") pod \"a71558c8-c93c-4383-9db1-041926890e0b\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") "
Jul 07 15:10:42.404879 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.404407    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a71558c8-c93c-4383-9db1-041926890e0b-cabundle-cert\") pod \"a71558c8-c93c-4383-9db1-041926890e0b\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") "
Jul 07 15:10:42.404879 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.404478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a71558c8-c93c-4383-9db1-041926890e0b-kube-rbac-proxy-sar-config\") pod \"a71558c8-c93c-4383-9db1-041926890e0b\" (UID: \"a71558c8-c93c-4383-9db1-041926890e0b\") "
Jul 07 15:10:42.404879 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.404695    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71558c8-c93c-4383-9db1-041926890e0b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a71558c8-c93c-4383-9db1-041926890e0b" (UID: "a71558c8-c93c-4383-9db1-041926890e0b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:10:42.404879 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.404802    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71558c8-c93c-4383-9db1-041926890e0b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a71558c8-c93c-4383-9db1-041926890e0b" (UID: "a71558c8-c93c-4383-9db1-041926890e0b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:10:42.405039 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.404964    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71558c8-c93c-4383-9db1-041926890e0b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a71558c8-c93c-4383-9db1-041926890e0b" (UID: "a71558c8-c93c-4383-9db1-041926890e0b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:10:42.406962 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.406940    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a71558c8-c93c-4383-9db1-041926890e0b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a71558c8-c93c-4383-9db1-041926890e0b" (UID: "a71558c8-c93c-4383-9db1-041926890e0b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:10:42.407106 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.407022    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71558c8-c93c-4383-9db1-041926890e0b-kube-api-access-nngdx" (OuterVolumeSpecName: "kube-api-access-nngdx") pod "a71558c8-c93c-4383-9db1-041926890e0b" (UID: "a71558c8-c93c-4383-9db1-041926890e0b"). InnerVolumeSpecName "kube-api-access-nngdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:10:42.410648 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.410624    2581 generic.go:356] "Generic (PLEG): container finished" podID="a71558c8-c93c-4383-9db1-041926890e0b" containerID="9925fbec845ad7a0f7f774480ff630b3e79378e01b629133b66b51c93650f33c" exitCode=0
Jul 07 15:10:42.410773 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.410677    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"
Jul 07 15:10:42.410773 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.410703    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" event={"ID":"a71558c8-c93c-4383-9db1-041926890e0b","Type":"ContainerDied","Data":"9925fbec845ad7a0f7f774480ff630b3e79378e01b629133b66b51c93650f33c"}
Jul 07 15:10:42.410896 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.410777    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz" event={"ID":"a71558c8-c93c-4383-9db1-041926890e0b","Type":"ContainerDied","Data":"879ead6184961b2a08613ad6b8491ca8b9b59b168a536358bf504850870b1ac4"}
Jul 07 15:10:42.410896 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.410822    2581 scope.go:117] "RemoveContainer" containerID="5dba4b1382a4a9b40fd98129d0b0a11d9d6164904d085aaa1c90719b59893323"
Jul 07 15:10:42.413655 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.413633    2581 generic.go:356] "Generic (PLEG): container finished" podID="3cd44159-004a-40b3-9a1e-90596e648f90" containerID="d35e89bbd1abfbba934448cbb56af17af0e8d55a8758dad9b0c67940134b35ac" exitCode=0
Jul 07 15:10:42.413762 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.413705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" event={"ID":"3cd44159-004a-40b3-9a1e-90596e648f90","Type":"ContainerDied","Data":"d35e89bbd1abfbba934448cbb56af17af0e8d55a8758dad9b0c67940134b35ac"}
Jul 07 15:10:42.419526 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.419505    2581 scope.go:117] "RemoveContainer" containerID="9925fbec845ad7a0f7f774480ff630b3e79378e01b629133b66b51c93650f33c"
Jul 07 15:10:42.427262 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.427242    2581 scope.go:117] "RemoveContainer" containerID="324e250de90ecf143ba6e6a9c3a1fd6f8b8975d2d6b9536019c58b4976cb25b8"
Jul 07 15:10:42.440795 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.440776    2581 scope.go:117] "RemoveContainer" containerID="5dba4b1382a4a9b40fd98129d0b0a11d9d6164904d085aaa1c90719b59893323"
Jul 07 15:10:42.441153 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:10:42.441126    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dba4b1382a4a9b40fd98129d0b0a11d9d6164904d085aaa1c90719b59893323\": container with ID starting with 5dba4b1382a4a9b40fd98129d0b0a11d9d6164904d085aaa1c90719b59893323 not found: ID does not exist" containerID="5dba4b1382a4a9b40fd98129d0b0a11d9d6164904d085aaa1c90719b59893323"
Jul 07 15:10:42.441211 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.441161    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dba4b1382a4a9b40fd98129d0b0a11d9d6164904d085aaa1c90719b59893323"} err="failed to get container status \"5dba4b1382a4a9b40fd98129d0b0a11d9d6164904d085aaa1c90719b59893323\": rpc error: code = NotFound desc = could not find container \"5dba4b1382a4a9b40fd98129d0b0a11d9d6164904d085aaa1c90719b59893323\": container with ID starting with 5dba4b1382a4a9b40fd98129d0b0a11d9d6164904d085aaa1c90719b59893323 not found: ID does not exist"
Jul 07 15:10:42.441211 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.441179    2581 scope.go:117] "RemoveContainer" containerID="9925fbec845ad7a0f7f774480ff630b3e79378e01b629133b66b51c93650f33c"
Jul 07 15:10:42.441473 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:10:42.441447    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9925fbec845ad7a0f7f774480ff630b3e79378e01b629133b66b51c93650f33c\": container with ID starting with 9925fbec845ad7a0f7f774480ff630b3e79378e01b629133b66b51c93650f33c not found: ID does not exist" containerID="9925fbec845ad7a0f7f774480ff630b3e79378e01b629133b66b51c93650f33c"
Jul 07 15:10:42.441545 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.441483    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9925fbec845ad7a0f7f774480ff630b3e79378e01b629133b66b51c93650f33c"} err="failed to get container status \"9925fbec845ad7a0f7f774480ff630b3e79378e01b629133b66b51c93650f33c\": rpc error: code = NotFound desc = could not find container \"9925fbec845ad7a0f7f774480ff630b3e79378e01b629133b66b51c93650f33c\": container with ID starting with 9925fbec845ad7a0f7f774480ff630b3e79378e01b629133b66b51c93650f33c not found: ID does not exist"
Jul 07 15:10:42.441545 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.441507    2581 scope.go:117] "RemoveContainer" containerID="324e250de90ecf143ba6e6a9c3a1fd6f8b8975d2d6b9536019c58b4976cb25b8"
Jul 07 15:10:42.441789 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:10:42.441773    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324e250de90ecf143ba6e6a9c3a1fd6f8b8975d2d6b9536019c58b4976cb25b8\": container with ID starting with 324e250de90ecf143ba6e6a9c3a1fd6f8b8975d2d6b9536019c58b4976cb25b8 not found: ID does not exist" containerID="324e250de90ecf143ba6e6a9c3a1fd6f8b8975d2d6b9536019c58b4976cb25b8"
Jul 07 15:10:42.441850 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.441798    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324e250de90ecf143ba6e6a9c3a1fd6f8b8975d2d6b9536019c58b4976cb25b8"} err="failed to get container status \"324e250de90ecf143ba6e6a9c3a1fd6f8b8975d2d6b9536019c58b4976cb25b8\": rpc error: code = NotFound desc = could not find container \"324e250de90ecf143ba6e6a9c3a1fd6f8b8975d2d6b9536019c58b4976cb25b8\": container with ID starting with 324e250de90ecf143ba6e6a9c3a1fd6f8b8975d2d6b9536019c58b4976cb25b8 not found: ID does not exist"
Jul 07 15:10:42.453455 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.453422    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"]
Jul 07 15:10:42.457538 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.457513    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3a4a-predictor-6f4c9d8749-65zpz"]
Jul 07 15:10:42.505544 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.505520    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a71558c8-c93c-4383-9db1-041926890e0b-cabundle-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:10:42.505669 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.505548    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a71558c8-c93c-4383-9db1-041926890e0b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:10:42.505669 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.505562    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nngdx\" (UniqueName: \"kubernetes.io/projected/a71558c8-c93c-4383-9db1-041926890e0b-kube-api-access-nngdx\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:10:42.505669 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.505577    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a71558c8-c93c-4383-9db1-041926890e0b-kserve-provision-location\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:10:42.505669 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.505591    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a71558c8-c93c-4383-9db1-041926890e0b-proxy-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:10:42.933467 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.933431    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" path="/var/lib/kubelet/pods/8ec8ada2-35b3-483d-be07-e4fe373dd681/volumes"
Jul 07 15:10:42.934008 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:42.933988    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a71558c8-c93c-4383-9db1-041926890e0b" path="/var/lib/kubelet/pods/a71558c8-c93c-4383-9db1-041926890e0b/volumes"
Jul 07 15:10:43.420835 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:43.420798    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" event={"ID":"3cd44159-004a-40b3-9a1e-90596e648f90","Type":"ContainerStarted","Data":"eb488c6224410757623aeccc193944a8c40be6def61281caf963c313000a9e52"}
Jul 07 15:10:43.421307 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:43.420879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" event={"ID":"3cd44159-004a-40b3-9a1e-90596e648f90","Type":"ContainerStarted","Data":"dd93b871c407158ae143ef56e72bb82ee90dbba33c40b9eb6f1f219c799f3ef0"}
Jul 07 15:10:43.421307 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:43.421219    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:43.421445 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:43.421377    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:43.422475 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:43.422452    2581 generic.go:356] "Generic (PLEG): container finished" podID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerID="f67f7cc6bfdd9c665dde3eea015110cb7a478cc8003f92261983c3efb9114911" exitCode=0
Jul 07 15:10:43.422601 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:43.422489    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" event={"ID":"26714cf3-2b63-4b65-90c4-6b71822b067d","Type":"ContainerDied","Data":"f67f7cc6bfdd9c665dde3eea015110cb7a478cc8003f92261983c3efb9114911"}
Jul 07 15:10:43.422751 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:43.422729    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:10:43.446139 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:43.446073    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" podStartSLOduration=2.546826186 podStartE2EDuration="6.446052081s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.899225895s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:10:43.4460185 +0000 UTC m=+805.178630821" watchObservedRunningTime="2026-07-07 15:10:43.446052081 +0000 UTC m=+805.178664404"
Jul 07 15:10:44.432103 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:44.432066    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" event={"ID":"26714cf3-2b63-4b65-90c4-6b71822b067d","Type":"ContainerStarted","Data":"fcd09336410251c621c0356d12c7aa9a464476a28638a090f621e8d356b32b31"}
Jul 07 15:10:44.432526 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:44.432151    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" event={"ID":"26714cf3-2b63-4b65-90c4-6b71822b067d","Type":"ContainerStarted","Data":"d3b2df26ea9b5635e62cb92304b76be0f7d08809bfdf0ebc288e208ca9e7dc7c"}
Jul 07 15:10:44.432585 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:44.432522    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:10:44.432635 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:44.432537    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:44.432698 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:44.432682    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:44.433930 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:44.433905    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:10:44.475285 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:44.475236    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" podStartSLOduration=3.537044543 podStartE2EDuration="7.475222041s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.938177498s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:10:44.474879028 +0000 UTC m=+806.207491348" watchObservedRunningTime="2026-07-07 15:10:44.475222041 +0000 UTC m=+806.207834361"
Jul 07 15:10:45.437623 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:45.437583    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:10:46.442174 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:46.442135    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:10:49.437089 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:49.437055    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:10:49.437616 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:49.437581    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:10:51.447523 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:51.447495    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:10:51.448013 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:51.447987    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:10:59.438366 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:10:59.438316    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:11:01.448293 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:11:01.448258    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:11:09.437725 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:11:09.437682    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:11:11.448063 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:11:11.448016    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:11:19.437620 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:11:19.437571    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:11:21.448445 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:11:21.448397    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:11:29.437750 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:11:29.437704    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:11:31.448661 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:11:31.448622    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:11:39.437659 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:11:39.437622    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:11:41.448366 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:11:41.448299    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:11:49.437553 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:11:49.437512    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:11:51.448525 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:11:51.448495    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:11:59.438564 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:11:59.438527    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:12:27.672051 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.671960    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"]
Jul 07 15:12:27.672611 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672591    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="storage-initializer"
Jul 07 15:12:27.672697 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672613    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="storage-initializer"
Jul 07 15:12:27.672697 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672627    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kserve-container"
Jul 07 15:12:27.672697 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672635    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kserve-container"
Jul 07 15:12:27.672697 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672647    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kube-rbac-proxy"
Jul 07 15:12:27.672697 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672656    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kube-rbac-proxy"
Jul 07 15:12:27.672697 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672668    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="storage-initializer"
Jul 07 15:12:27.672697 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672677    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="storage-initializer"
Jul 07 15:12:27.672697 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672699    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container"
Jul 07 15:12:27.673075 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672707    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container"
Jul 07 15:12:27.673075 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672723    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kube-rbac-proxy"
Jul 07 15:12:27.673075 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672733    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kube-rbac-proxy"
Jul 07 15:12:27.673075 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672831    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kserve-container"
Jul 07 15:12:27.673075 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672843    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kserve-container"
Jul 07 15:12:27.673075 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672857    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a71558c8-c93c-4383-9db1-041926890e0b" containerName="kube-rbac-proxy"
Jul 07 15:12:27.673075 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.672866    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec8ada2-35b3-483d-be07-e4fe373dd681" containerName="kube-rbac-proxy"
Jul 07 15:12:27.675265 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.675243    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:12:27.678190 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.678166    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-c27ac-predictor-serving-cert\""
Jul 07 15:12:27.678190 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.678184    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-c27ac-kube-rbac-proxy-sar-config\""
Jul 07 15:12:27.686812 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.686779    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"]
Jul 07 15:12:27.696086 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.696063    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"]
Jul 07 15:12:27.696319 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.696297    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kserve-container" containerID="cri-o://dd93b871c407158ae143ef56e72bb82ee90dbba33c40b9eb6f1f219c799f3ef0" gracePeriod=30
Jul 07 15:12:27.696435 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.696410    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kube-rbac-proxy" containerID="cri-o://eb488c6224410757623aeccc193944a8c40be6def61281caf963c313000a9e52" gracePeriod=30
Jul 07 15:12:27.736602 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.736568    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"]
Jul 07 15:12:27.736836 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.736796    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kserve-container" containerID="cri-o://d3b2df26ea9b5635e62cb92304b76be0f7d08809bfdf0ebc288e208ca9e7dc7c" gracePeriod=30
Jul 07 15:12:27.736894 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.736832    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kube-rbac-proxy" containerID="cri-o://fcd09336410251c621c0356d12c7aa9a464476a28638a090f621e8d356b32b31" gracePeriod=30
Jul 07 15:12:27.741248 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.741219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68a8729c-ac00-4e21-aee6-b769b50f5a87-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr\" (UID: \"68a8729c-ac00-4e21-aee6-b769b50f5a87\") " pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:12:27.741392 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.741302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wshqt\" (UniqueName: \"kubernetes.io/projected/68a8729c-ac00-4e21-aee6-b769b50f5a87-kube-api-access-wshqt\") pod \"message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr\" (UID: \"68a8729c-ac00-4e21-aee6-b769b50f5a87\") " pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:12:27.741392 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.741330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68a8729c-ac00-4e21-aee6-b769b50f5a87-proxy-tls\") pod \"message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr\" (UID: \"68a8729c-ac00-4e21-aee6-b769b50f5a87\") " pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:12:27.841859 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.841832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wshqt\" (UniqueName: \"kubernetes.io/projected/68a8729c-ac00-4e21-aee6-b769b50f5a87-kube-api-access-wshqt\") pod \"message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr\" (UID: \"68a8729c-ac00-4e21-aee6-b769b50f5a87\") " pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:12:27.841935 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.841868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68a8729c-ac00-4e21-aee6-b769b50f5a87-proxy-tls\") pod \"message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr\" (UID: \"68a8729c-ac00-4e21-aee6-b769b50f5a87\") " pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:12:27.841935 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.841930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68a8729c-ac00-4e21-aee6-b769b50f5a87-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr\" (UID: \"68a8729c-ac00-4e21-aee6-b769b50f5a87\") " pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:12:27.842579 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.842556    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68a8729c-ac00-4e21-aee6-b769b50f5a87-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr\" (UID: \"68a8729c-ac00-4e21-aee6-b769b50f5a87\") " pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:12:27.844333 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.844305    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68a8729c-ac00-4e21-aee6-b769b50f5a87-proxy-tls\") pod \"message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr\" (UID: \"68a8729c-ac00-4e21-aee6-b769b50f5a87\") " pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:12:27.850304 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.850280    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wshqt\" (UniqueName: \"kubernetes.io/projected/68a8729c-ac00-4e21-aee6-b769b50f5a87-kube-api-access-wshqt\") pod \"message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr\" (UID: \"68a8729c-ac00-4e21-aee6-b769b50f5a87\") " pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:12:27.889042 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.889002    2581 generic.go:356] "Generic (PLEG): container finished" podID="3cd44159-004a-40b3-9a1e-90596e648f90" containerID="eb488c6224410757623aeccc193944a8c40be6def61281caf963c313000a9e52" exitCode=2
Jul 07 15:12:27.889222 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.889069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" event={"ID":"3cd44159-004a-40b3-9a1e-90596e648f90","Type":"ContainerDied","Data":"eb488c6224410757623aeccc193944a8c40be6def61281caf963c313000a9e52"}
Jul 07 15:12:27.891007 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.890986    2581 generic.go:356] "Generic (PLEG): container finished" podID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerID="fcd09336410251c621c0356d12c7aa9a464476a28638a090f621e8d356b32b31" exitCode=2
Jul 07 15:12:27.891128 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.891041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" event={"ID":"26714cf3-2b63-4b65-90c4-6b71822b067d","Type":"ContainerDied","Data":"fcd09336410251c621c0356d12c7aa9a464476a28638a090f621e8d356b32b31"}
Jul 07 15:12:27.987602 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:27.987503    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:12:28.123084 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:28.122997    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"]
Jul 07 15:12:28.125631 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:12:28.125597    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a8729c_ac00_4e21_aee6_b769b50f5a87.slice/crio-08212a666aa44fc7c950570fb47d95ae0898da55063884b8958d53d8e060275a WatchSource:0}: Error finding container 08212a666aa44fc7c950570fb47d95ae0898da55063884b8958d53d8e060275a: Status 404 returned error can't find the container with id 08212a666aa44fc7c950570fb47d95ae0898da55063884b8958d53d8e060275a
Jul 07 15:12:28.128077 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:28.128062    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 15:12:28.896596 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:28.896563    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr" event={"ID":"68a8729c-ac00-4e21-aee6-b769b50f5a87","Type":"ContainerStarted","Data":"08212a666aa44fc7c950570fb47d95ae0898da55063884b8958d53d8e060275a"}
Jul 07 15:12:29.433557 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:29.433521    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 15:12:29.437996 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:29.437963    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:12:29.902336 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:29.902296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr" event={"ID":"68a8729c-ac00-4e21-aee6-b769b50f5a87","Type":"ContainerStarted","Data":"07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a"}
Jul 07 15:12:29.902793 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:29.902373    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:12:29.902793 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:29.902395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr" event={"ID":"68a8729c-ac00-4e21-aee6-b769b50f5a87","Type":"ContainerStarted","Data":"f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0"}
Jul 07 15:12:29.902793 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:29.902416    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:12:29.904208 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:29.904189    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:12:29.925701 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:29.925643    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr" podStartSLOduration=1.481796112 podStartE2EDuration="2.925622813s" totalImagesPullingTime="1.443826701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:12:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:12:29.922003639 +0000 UTC m=+911.654615961" watchObservedRunningTime="2026-07-07 15:12:29.925622813 +0000 UTC m=+911.658235134"
Jul 07 15:12:31.442845 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.442796    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 07 15:12:31.448132 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.448105    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:12:31.685541 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.685515    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:12:31.781149 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.781056    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26714cf3-2b63-4b65-90c4-6b71822b067d-kube-rbac-proxy-sar-config\") pod \"26714cf3-2b63-4b65-90c4-6b71822b067d\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") "
Jul 07 15:12:31.781295 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.781162    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dskl5\" (UniqueName: \"kubernetes.io/projected/26714cf3-2b63-4b65-90c4-6b71822b067d-kube-api-access-dskl5\") pod \"26714cf3-2b63-4b65-90c4-6b71822b067d\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") "
Jul 07 15:12:31.781295 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.781245    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26714cf3-2b63-4b65-90c4-6b71822b067d-kserve-provision-location\") pod \"26714cf3-2b63-4b65-90c4-6b71822b067d\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") "
Jul 07 15:12:31.781295 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.781274    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26714cf3-2b63-4b65-90c4-6b71822b067d-cabundle-cert\") pod \"26714cf3-2b63-4b65-90c4-6b71822b067d\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") "
Jul 07 15:12:31.781460 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.781325    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26714cf3-2b63-4b65-90c4-6b71822b067d-proxy-tls\") pod \"26714cf3-2b63-4b65-90c4-6b71822b067d\" (UID: \"26714cf3-2b63-4b65-90c4-6b71822b067d\") "
Jul 07 15:12:31.781629 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.781604    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26714cf3-2b63-4b65-90c4-6b71822b067d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26714cf3-2b63-4b65-90c4-6b71822b067d" (UID: "26714cf3-2b63-4b65-90c4-6b71822b067d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:12:31.781713 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.781606    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26714cf3-2b63-4b65-90c4-6b71822b067d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26714cf3-2b63-4b65-90c4-6b71822b067d" (UID: "26714cf3-2b63-4b65-90c4-6b71822b067d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:12:31.781761 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.781708    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26714cf3-2b63-4b65-90c4-6b71822b067d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26714cf3-2b63-4b65-90c4-6b71822b067d" (UID: "26714cf3-2b63-4b65-90c4-6b71822b067d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:12:31.783343 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.783324    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26714cf3-2b63-4b65-90c4-6b71822b067d-kube-api-access-dskl5" (OuterVolumeSpecName: "kube-api-access-dskl5") pod "26714cf3-2b63-4b65-90c4-6b71822b067d" (UID: "26714cf3-2b63-4b65-90c4-6b71822b067d"). InnerVolumeSpecName "kube-api-access-dskl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:12:31.783431 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.783378    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26714cf3-2b63-4b65-90c4-6b71822b067d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26714cf3-2b63-4b65-90c4-6b71822b067d" (UID: "26714cf3-2b63-4b65-90c4-6b71822b067d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:12:31.882109 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.882072    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dskl5\" (UniqueName: \"kubernetes.io/projected/26714cf3-2b63-4b65-90c4-6b71822b067d-kube-api-access-dskl5\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:12:31.882109 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.882103    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26714cf3-2b63-4b65-90c4-6b71822b067d-kserve-provision-location\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:12:31.882109 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.882112    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26714cf3-2b63-4b65-90c4-6b71822b067d-cabundle-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:12:31.882109 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.882122    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26714cf3-2b63-4b65-90c4-6b71822b067d-proxy-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:12:31.882423 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.882130    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26714cf3-2b63-4b65-90c4-6b71822b067d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:12:31.913106 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.913076    2581 generic.go:356] "Generic (PLEG): container finished" podID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerID="d3b2df26ea9b5635e62cb92304b76be0f7d08809bfdf0ebc288e208ca9e7dc7c" exitCode=0
Jul 07 15:12:31.913253 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.913134    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"
Jul 07 15:12:31.913253 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.913157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" event={"ID":"26714cf3-2b63-4b65-90c4-6b71822b067d","Type":"ContainerDied","Data":"d3b2df26ea9b5635e62cb92304b76be0f7d08809bfdf0ebc288e208ca9e7dc7c"}
Jul 07 15:12:31.913253 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.913211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq" event={"ID":"26714cf3-2b63-4b65-90c4-6b71822b067d","Type":"ContainerDied","Data":"afd7d873a97dba74eb58363770b77ee1a5cb967bac31e0530573db7401076d1f"}
Jul 07 15:12:31.913253 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.913237    2581 scope.go:117] "RemoveContainer" containerID="fcd09336410251c621c0356d12c7aa9a464476a28638a090f621e8d356b32b31"
Jul 07 15:12:31.922069 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.922050    2581 scope.go:117] "RemoveContainer" containerID="d3b2df26ea9b5635e62cb92304b76be0f7d08809bfdf0ebc288e208ca9e7dc7c"
Jul 07 15:12:31.929898 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.929881    2581 scope.go:117] "RemoveContainer" containerID="f67f7cc6bfdd9c665dde3eea015110cb7a478cc8003f92261983c3efb9114911"
Jul 07 15:12:31.936103 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.936077    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"]
Jul 07 15:12:31.938172 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.938155    2581 scope.go:117] "RemoveContainer" containerID="fcd09336410251c621c0356d12c7aa9a464476a28638a090f621e8d356b32b31"
Jul 07 15:12:31.938536 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:12:31.938519    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd09336410251c621c0356d12c7aa9a464476a28638a090f621e8d356b32b31\": container with ID starting with fcd09336410251c621c0356d12c7aa9a464476a28638a090f621e8d356b32b31 not found: ID does not exist" containerID="fcd09336410251c621c0356d12c7aa9a464476a28638a090f621e8d356b32b31"
Jul 07 15:12:31.938603 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.938543    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd09336410251c621c0356d12c7aa9a464476a28638a090f621e8d356b32b31"} err="failed to get container status \"fcd09336410251c621c0356d12c7aa9a464476a28638a090f621e8d356b32b31\": rpc error: code = NotFound desc = could not find container \"fcd09336410251c621c0356d12c7aa9a464476a28638a090f621e8d356b32b31\": container with ID starting with fcd09336410251c621c0356d12c7aa9a464476a28638a090f621e8d356b32b31 not found: ID does not exist"
Jul 07 15:12:31.938603 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.938568    2581 scope.go:117] "RemoveContainer" containerID="d3b2df26ea9b5635e62cb92304b76be0f7d08809bfdf0ebc288e208ca9e7dc7c"
Jul 07 15:12:31.938809 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:12:31.938794    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b2df26ea9b5635e62cb92304b76be0f7d08809bfdf0ebc288e208ca9e7dc7c\": container with ID starting with d3b2df26ea9b5635e62cb92304b76be0f7d08809bfdf0ebc288e208ca9e7dc7c not found: ID does not exist" containerID="d3b2df26ea9b5635e62cb92304b76be0f7d08809bfdf0ebc288e208ca9e7dc7c"
Jul 07 15:12:31.938856 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.938821    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b2df26ea9b5635e62cb92304b76be0f7d08809bfdf0ebc288e208ca9e7dc7c"} err="failed to get container status \"d3b2df26ea9b5635e62cb92304b76be0f7d08809bfdf0ebc288e208ca9e7dc7c\": rpc error: code = NotFound desc = could not find container \"d3b2df26ea9b5635e62cb92304b76be0f7d08809bfdf0ebc288e208ca9e7dc7c\": container with ID starting with d3b2df26ea9b5635e62cb92304b76be0f7d08809bfdf0ebc288e208ca9e7dc7c not found: ID does not exist"
Jul 07 15:12:31.938856 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.938833    2581 scope.go:117] "RemoveContainer" containerID="f67f7cc6bfdd9c665dde3eea015110cb7a478cc8003f92261983c3efb9114911"
Jul 07 15:12:31.939046 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:12:31.939030    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67f7cc6bfdd9c665dde3eea015110cb7a478cc8003f92261983c3efb9114911\": container with ID starting with f67f7cc6bfdd9c665dde3eea015110cb7a478cc8003f92261983c3efb9114911 not found: ID does not exist" containerID="f67f7cc6bfdd9c665dde3eea015110cb7a478cc8003f92261983c3efb9114911"
Jul 07 15:12:31.939088 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.939047    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67f7cc6bfdd9c665dde3eea015110cb7a478cc8003f92261983c3efb9114911"} err="failed to get container status \"f67f7cc6bfdd9c665dde3eea015110cb7a478cc8003f92261983c3efb9114911\": rpc error: code = NotFound desc = could not find container \"f67f7cc6bfdd9c665dde3eea015110cb7a478cc8003f92261983c3efb9114911\": container with ID starting with f67f7cc6bfdd9c665dde3eea015110cb7a478cc8003f92261983c3efb9114911 not found: ID does not exist"
Jul 07 15:12:31.943093 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:31.943072    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0f7d-predictor-845db67fd5-rctvq"]
Jul 07 15:12:32.748837 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.748812    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:12:32.791242 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.791162    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cd44159-004a-40b3-9a1e-90596e648f90-proxy-tls\") pod \"3cd44159-004a-40b3-9a1e-90596e648f90\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") "
Jul 07 15:12:32.791242 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.791202    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cd44159-004a-40b3-9a1e-90596e648f90-kube-rbac-proxy-sar-config\") pod \"3cd44159-004a-40b3-9a1e-90596e648f90\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") "
Jul 07 15:12:32.791439 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.791399    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd44159-004a-40b3-9a1e-90596e648f90-kserve-provision-location\") pod \"3cd44159-004a-40b3-9a1e-90596e648f90\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") "
Jul 07 15:12:32.791477 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.791440    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cd44159-004a-40b3-9a1e-90596e648f90-cabundle-cert\") pod \"3cd44159-004a-40b3-9a1e-90596e648f90\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") "
Jul 07 15:12:32.791515 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.791473    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd9bg\" (UniqueName: \"kubernetes.io/projected/3cd44159-004a-40b3-9a1e-90596e648f90-kube-api-access-dd9bg\") pod \"3cd44159-004a-40b3-9a1e-90596e648f90\" (UID: \"3cd44159-004a-40b3-9a1e-90596e648f90\") "
Jul 07 15:12:32.791590 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.791561    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd44159-004a-40b3-9a1e-90596e648f90-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3cd44159-004a-40b3-9a1e-90596e648f90" (UID: "3cd44159-004a-40b3-9a1e-90596e648f90"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:12:32.791748 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.791725    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd44159-004a-40b3-9a1e-90596e648f90-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3cd44159-004a-40b3-9a1e-90596e648f90" (UID: "3cd44159-004a-40b3-9a1e-90596e648f90"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:12:32.791748 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.791740    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd44159-004a-40b3-9a1e-90596e648f90-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3cd44159-004a-40b3-9a1e-90596e648f90" (UID: "3cd44159-004a-40b3-9a1e-90596e648f90"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:12:32.791854 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.791778    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cd44159-004a-40b3-9a1e-90596e648f90-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:12:32.793382 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.793359    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd44159-004a-40b3-9a1e-90596e648f90-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3cd44159-004a-40b3-9a1e-90596e648f90" (UID: "3cd44159-004a-40b3-9a1e-90596e648f90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:12:32.793469 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.793388    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd44159-004a-40b3-9a1e-90596e648f90-kube-api-access-dd9bg" (OuterVolumeSpecName: "kube-api-access-dd9bg") pod "3cd44159-004a-40b3-9a1e-90596e648f90" (UID: "3cd44159-004a-40b3-9a1e-90596e648f90"). InnerVolumeSpecName "kube-api-access-dd9bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:12:32.892947 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.892911    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cd44159-004a-40b3-9a1e-90596e648f90-proxy-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:12:32.892947 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.892942    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd44159-004a-40b3-9a1e-90596e648f90-kserve-provision-location\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:12:32.892947 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.892952    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cd44159-004a-40b3-9a1e-90596e648f90-cabundle-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:12:32.893218 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.892961    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd9bg\" (UniqueName: \"kubernetes.io/projected/3cd44159-004a-40b3-9a1e-90596e648f90-kube-api-access-dd9bg\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:12:32.919335 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.919283    2581 generic.go:356] "Generic (PLEG): container finished" podID="3cd44159-004a-40b3-9a1e-90596e648f90" containerID="dd93b871c407158ae143ef56e72bb82ee90dbba33c40b9eb6f1f219c799f3ef0" exitCode=0
Jul 07 15:12:32.919531 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.919369    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"
Jul 07 15:12:32.919531 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.919390    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" event={"ID":"3cd44159-004a-40b3-9a1e-90596e648f90","Type":"ContainerDied","Data":"dd93b871c407158ae143ef56e72bb82ee90dbba33c40b9eb6f1f219c799f3ef0"}
Jul 07 15:12:32.919531 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.919461    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl" event={"ID":"3cd44159-004a-40b3-9a1e-90596e648f90","Type":"ContainerDied","Data":"a74bde79593d0e240f1dab3848220537435146e6bee3c63b408234f7666279b7"}
Jul 07 15:12:32.919531 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.919493    2581 scope.go:117] "RemoveContainer" containerID="eb488c6224410757623aeccc193944a8c40be6def61281caf963c313000a9e52"
Jul 07 15:12:32.928040 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.928024    2581 scope.go:117] "RemoveContainer" containerID="dd93b871c407158ae143ef56e72bb82ee90dbba33c40b9eb6f1f219c799f3ef0"
Jul 07 15:12:32.932131 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.932107    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" path="/var/lib/kubelet/pods/26714cf3-2b63-4b65-90c4-6b71822b067d/volumes"
Jul 07 15:12:32.935760 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.935742    2581 scope.go:117] "RemoveContainer" containerID="d35e89bbd1abfbba934448cbb56af17af0e8d55a8758dad9b0c67940134b35ac"
Jul 07 15:12:32.941980 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.941958    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"]
Jul 07 15:12:32.943835 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.943764    2581 scope.go:117] "RemoveContainer" containerID="eb488c6224410757623aeccc193944a8c40be6def61281caf963c313000a9e52"
Jul 07 15:12:32.944065 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:12:32.944042    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb488c6224410757623aeccc193944a8c40be6def61281caf963c313000a9e52\": container with ID starting with eb488c6224410757623aeccc193944a8c40be6def61281caf963c313000a9e52 not found: ID does not exist" containerID="eb488c6224410757623aeccc193944a8c40be6def61281caf963c313000a9e52"
Jul 07 15:12:32.944120 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.944070    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb488c6224410757623aeccc193944a8c40be6def61281caf963c313000a9e52"} err="failed to get container status \"eb488c6224410757623aeccc193944a8c40be6def61281caf963c313000a9e52\": rpc error: code = NotFound desc = could not find container \"eb488c6224410757623aeccc193944a8c40be6def61281caf963c313000a9e52\": container with ID starting with eb488c6224410757623aeccc193944a8c40be6def61281caf963c313000a9e52 not found: ID does not exist"
Jul 07 15:12:32.944120 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.944087    2581 scope.go:117] "RemoveContainer" containerID="dd93b871c407158ae143ef56e72bb82ee90dbba33c40b9eb6f1f219c799f3ef0"
Jul 07 15:12:32.944327 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:12:32.944314    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd93b871c407158ae143ef56e72bb82ee90dbba33c40b9eb6f1f219c799f3ef0\": container with ID starting with dd93b871c407158ae143ef56e72bb82ee90dbba33c40b9eb6f1f219c799f3ef0 not found: ID does not exist" containerID="dd93b871c407158ae143ef56e72bb82ee90dbba33c40b9eb6f1f219c799f3ef0"
Jul 07 15:12:32.944397 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.944330    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd93b871c407158ae143ef56e72bb82ee90dbba33c40b9eb6f1f219c799f3ef0"} err="failed to get container status \"dd93b871c407158ae143ef56e72bb82ee90dbba33c40b9eb6f1f219c799f3ef0\": rpc error: code = NotFound desc = could not find container \"dd93b871c407158ae143ef56e72bb82ee90dbba33c40b9eb6f1f219c799f3ef0\": container with ID starting with dd93b871c407158ae143ef56e72bb82ee90dbba33c40b9eb6f1f219c799f3ef0 not found: ID does not exist"
Jul 07 15:12:32.944397 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.944343    2581 scope.go:117] "RemoveContainer" containerID="d35e89bbd1abfbba934448cbb56af17af0e8d55a8758dad9b0c67940134b35ac"
Jul 07 15:12:32.944628 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:12:32.944607    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35e89bbd1abfbba934448cbb56af17af0e8d55a8758dad9b0c67940134b35ac\": container with ID starting with d35e89bbd1abfbba934448cbb56af17af0e8d55a8758dad9b0c67940134b35ac not found: ID does not exist" containerID="d35e89bbd1abfbba934448cbb56af17af0e8d55a8758dad9b0c67940134b35ac"
Jul 07 15:12:32.944678 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.944635    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35e89bbd1abfbba934448cbb56af17af0e8d55a8758dad9b0c67940134b35ac"} err="failed to get container status \"d35e89bbd1abfbba934448cbb56af17af0e8d55a8758dad9b0c67940134b35ac\": rpc error: code = NotFound desc = could not find container \"d35e89bbd1abfbba934448cbb56af17af0e8d55a8758dad9b0c67940134b35ac\": container with ID starting with d35e89bbd1abfbba934448cbb56af17af0e8d55a8758dad9b0c67940134b35ac not found: ID does not exist"
Jul 07 15:12:32.945543 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:32.945522    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0f7d-predictor-58bbc5c47c-n67nl"]
Jul 07 15:12:34.932611 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:34.932571    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" path="/var/lib/kubelet/pods/3cd44159-004a-40b3-9a1e-90596e648f90/volumes"
Jul 07 15:12:36.918164 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:36.918130    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:12:37.669136 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669097    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"]
Jul 07 15:12:37.669563 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669542    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kube-rbac-proxy"
Jul 07 15:12:37.669640 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669566    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kube-rbac-proxy"
Jul 07 15:12:37.669640 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669585    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kserve-container"
Jul 07 15:12:37.669640 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669594    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kserve-container"
Jul 07 15:12:37.669640 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669633    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="storage-initializer"
Jul 07 15:12:37.669777 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669642    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="storage-initializer"
Jul 07 15:12:37.669777 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669655    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kserve-container"
Jul 07 15:12:37.669777 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669664    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kserve-container"
Jul 07 15:12:37.669777 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669676    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="storage-initializer"
Jul 07 15:12:37.669777 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669681    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="storage-initializer"
Jul 07 15:12:37.669777 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669687    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kube-rbac-proxy"
Jul 07 15:12:37.669777 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669692    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kube-rbac-proxy"
Jul 07 15:12:37.669777 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669760    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kserve-container"
Jul 07 15:12:37.669777 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669769    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cd44159-004a-40b3-9a1e-90596e648f90" containerName="kube-rbac-proxy"
Jul 07 15:12:37.669777 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669776    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kserve-container"
Jul 07 15:12:37.669777 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.669782    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="26714cf3-2b63-4b65-90c4-6b71822b067d" containerName="kube-rbac-proxy"
Jul 07 15:12:37.672991 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.672968    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:37.675891 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.675866    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-c27ac-predictor-serving-cert\""
Jul 07 15:12:37.676006 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.675870    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 15:12:37.676006 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.675908    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-c27ac-kube-rbac-proxy-sar-config\""
Jul 07 15:12:37.676178 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.676165    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 15:12:37.686037 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.686013    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"]
Jul 07 15:12:37.733819 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.733789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066b718a-05bd-447a-b270-d84635d62900-proxy-tls\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:37.733993 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.733830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/066b718a-05bd-447a-b270-d84635d62900-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:37.733993 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.733855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066b718a-05bd-447a-b270-d84635d62900-cabundle-cert\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:37.733993 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.733952    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066b718a-05bd-447a-b270-d84635d62900-kserve-provision-location\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:37.734146 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.733995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zmpg\" (UniqueName: \"kubernetes.io/projected/066b718a-05bd-447a-b270-d84635d62900-kube-api-access-4zmpg\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:37.835297 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.835261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066b718a-05bd-447a-b270-d84635d62900-proxy-tls\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:37.835462 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.835306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/066b718a-05bd-447a-b270-d84635d62900-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:37.835462 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.835328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066b718a-05bd-447a-b270-d84635d62900-cabundle-cert\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:37.835462 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.835392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066b718a-05bd-447a-b270-d84635d62900-kserve-provision-location\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:37.835462 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.835428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zmpg\" (UniqueName: \"kubernetes.io/projected/066b718a-05bd-447a-b270-d84635d62900-kube-api-access-4zmpg\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:37.835689 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:12:37.835475    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-serving-cert: secret "isvc-logger-raw-c27ac-predictor-serving-cert" not found
Jul 07 15:12:37.835689 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:12:37.835564    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066b718a-05bd-447a-b270-d84635d62900-proxy-tls podName:066b718a-05bd-447a-b270-d84635d62900 nodeName:}" failed. No retries permitted until 2026-07-07 15:12:38.335542892 +0000 UTC m=+920.068155203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/066b718a-05bd-447a-b270-d84635d62900-proxy-tls") pod "isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" (UID: "066b718a-05bd-447a-b270-d84635d62900") : secret "isvc-logger-raw-c27ac-predictor-serving-cert" not found
Jul 07 15:12:37.835875 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.835854    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066b718a-05bd-447a-b270-d84635d62900-kserve-provision-location\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:37.835991 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.835973    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/066b718a-05bd-447a-b270-d84635d62900-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:37.836060 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.836043    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066b718a-05bd-447a-b270-d84635d62900-cabundle-cert\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:37.846418 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:37.846391    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zmpg\" (UniqueName: \"kubernetes.io/projected/066b718a-05bd-447a-b270-d84635d62900-kube-api-access-4zmpg\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:38.340221 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:38.340166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066b718a-05bd-447a-b270-d84635d62900-proxy-tls\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:38.342545 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:38.342521    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066b718a-05bd-447a-b270-d84635d62900-proxy-tls\") pod \"isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:38.583713 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:38.583671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:38.720467 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:38.720390    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"]
Jul 07 15:12:38.722823 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:12:38.722795    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066b718a_05bd_447a_b270_d84635d62900.slice/crio-61253b3e577bf7728d710576eeb9ce06c1cc8c0e5ec7854232ec09b9ce175ccd WatchSource:0}: Error finding container 61253b3e577bf7728d710576eeb9ce06c1cc8c0e5ec7854232ec09b9ce175ccd: Status 404 returned error can't find the container with id 61253b3e577bf7728d710576eeb9ce06c1cc8c0e5ec7854232ec09b9ce175ccd
Jul 07 15:12:38.950423 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:38.950379    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" event={"ID":"066b718a-05bd-447a-b270-d84635d62900","Type":"ContainerStarted","Data":"58c4d29783feb2383c1948538a6c1f507afb47c1586770944724797ef20dccd7"}
Jul 07 15:12:38.950684 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:38.950455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" event={"ID":"066b718a-05bd-447a-b270-d84635d62900","Type":"ContainerStarted","Data":"61253b3e577bf7728d710576eeb9ce06c1cc8c0e5ec7854232ec09b9ce175ccd"}
Jul 07 15:12:42.972914 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:42.972881    2581 generic.go:356] "Generic (PLEG): container finished" podID="066b718a-05bd-447a-b270-d84635d62900" containerID="58c4d29783feb2383c1948538a6c1f507afb47c1586770944724797ef20dccd7" exitCode=0
Jul 07 15:12:42.973431 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:42.972945    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" event={"ID":"066b718a-05bd-447a-b270-d84635d62900","Type":"ContainerDied","Data":"58c4d29783feb2383c1948538a6c1f507afb47c1586770944724797ef20dccd7"}
Jul 07 15:12:43.979470 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:43.979435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" event={"ID":"066b718a-05bd-447a-b270-d84635d62900","Type":"ContainerStarted","Data":"086f06834d25361ede60de872e3907746aea383f26ae5140d82b2c0281fd7d3c"}
Jul 07 15:12:43.979953 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:43.979513    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" event={"ID":"066b718a-05bd-447a-b270-d84635d62900","Type":"ContainerStarted","Data":"a06c8da10e492b1d005cb8970a897719d913f68d83bee15d057c9b568f287d8b"}
Jul 07 15:12:43.979953 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:43.979549    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" event={"ID":"066b718a-05bd-447a-b270-d84635d62900","Type":"ContainerStarted","Data":"ff632fd676930d0eac480283aa782c48c65bb4d62c8bb72a63730d7de2ec2822"}
Jul 07 15:12:43.979953 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:43.979793    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:44.005270 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:44.005214    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podStartSLOduration=3.215229223 podStartE2EDuration="7.005197679s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.789968456s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:12:44.002637979 +0000 UTC m=+925.735250299" watchObservedRunningTime="2026-07-07 15:12:44.005197679 +0000 UTC m=+925.737810000"
Jul 07 15:12:44.985301 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:44.985268    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:44.985752 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:44.985343    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:44.987217 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:44.987185    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:12:44.988235 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:44.988203    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:12:45.990408 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:45.990361    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:12:45.990834 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:45.990738    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:12:50.994913 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:50.994881    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:12:50.995497 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:50.995470    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:12:50.995763 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:12:50.995733    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:13:00.996225 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:13:00.996181    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:13:00.996710 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:13:00.996686    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:13:10.995940 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:13:10.995886    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:13:10.996413 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:13:10.996298    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:13:20.995921 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:13:20.995878    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:13:20.998327 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:13:20.996258    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:13:30.995635 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:13:30.995587    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:13:30.996192 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:13:30.995961    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:13:40.996265 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:13:40.996171    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:13:40.996725 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:13:40.996627    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:13:50.995830 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:13:50.995785    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:13:50.996250 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:13:50.996225    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:14:00.996242 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:00.996212    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:14:00.996702 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:00.996473    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:14:12.725131 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:12.725094    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr_68a8729c-ac00-4e21-aee6-b769b50f5a87/kserve-container/0.log"
Jul 07 15:14:13.028411 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.028318    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"]
Jul 07 15:14:13.028646 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.028621    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container" containerID="cri-o://ff632fd676930d0eac480283aa782c48c65bb4d62c8bb72a63730d7de2ec2822" gracePeriod=30
Jul 07 15:14:13.028737 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.028639    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kube-rbac-proxy" containerID="cri-o://a06c8da10e492b1d005cb8970a897719d913f68d83bee15d057c9b568f287d8b" gracePeriod=30
Jul 07 15:14:13.028737 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.028646    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent" containerID="cri-o://086f06834d25361ede60de872e3907746aea383f26ae5140d82b2c0281fd7d3c" gracePeriod=30
Jul 07 15:14:13.032657 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.032633    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"]
Jul 07 15:14:13.036868 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.036848    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.039713 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.039691    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-11363-predictor-serving-cert\""
Jul 07 15:14:13.039819 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.039724    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-11363-kube-rbac-proxy-sar-config\""
Jul 07 15:14:13.049990 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.049967    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"]
Jul 07 15:14:13.072934 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.072906    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"]
Jul 07 15:14:13.073232 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.073193    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr" podUID="68a8729c-ac00-4e21-aee6-b769b50f5a87" containerName="kserve-container" containerID="cri-o://f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0" gracePeriod=30
Jul 07 15:14:13.073382 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.073275    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr" podUID="68a8729c-ac00-4e21-aee6-b769b50f5a87" containerName="kube-rbac-proxy" containerID="cri-o://07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a" gracePeriod=30
Jul 07 15:14:13.152891 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.152853    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75d74119-b14c-49a4-b638-275bf78f6536-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.152891 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.152889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75d74119-b14c-49a4-b638-275bf78f6536-cabundle-cert\") pod \"isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.153094 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.152957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj6tj\" (UniqueName: \"kubernetes.io/projected/75d74119-b14c-49a4-b638-275bf78f6536-kube-api-access-gj6tj\") pod \"isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.153094 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.152995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75d74119-b14c-49a4-b638-275bf78f6536-proxy-tls\") pod \"isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.153094 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.153046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75d74119-b14c-49a4-b638-275bf78f6536-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.253616 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.253582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj6tj\" (UniqueName: \"kubernetes.io/projected/75d74119-b14c-49a4-b638-275bf78f6536-kube-api-access-gj6tj\") pod \"isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.253908 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.253873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75d74119-b14c-49a4-b638-275bf78f6536-proxy-tls\") pod \"isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.254027 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.253910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75d74119-b14c-49a4-b638-275bf78f6536-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.254027 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.253978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75d74119-b14c-49a4-b638-275bf78f6536-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.254419 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.254396    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75d74119-b14c-49a4-b638-275bf78f6536-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.254538 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.254464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75d74119-b14c-49a4-b638-275bf78f6536-cabundle-cert\") pod \"isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.254828 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.254793    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75d74119-b14c-49a4-b638-275bf78f6536-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.255013 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.254994    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75d74119-b14c-49a4-b638-275bf78f6536-cabundle-cert\") pod \"isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.256751 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.256620    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75d74119-b14c-49a4-b638-275bf78f6536-proxy-tls\") pod \"isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.264126 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.264100    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj6tj\" (UniqueName: \"kubernetes.io/projected/75d74119-b14c-49a4-b638-275bf78f6536-kube-api-access-gj6tj\") pod \"isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.312875 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.312848    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:14:13.355805 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.355773    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wshqt\" (UniqueName: \"kubernetes.io/projected/68a8729c-ac00-4e21-aee6-b769b50f5a87-kube-api-access-wshqt\") pod \"68a8729c-ac00-4e21-aee6-b769b50f5a87\" (UID: \"68a8729c-ac00-4e21-aee6-b769b50f5a87\") "
Jul 07 15:14:13.355995 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.355977    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68a8729c-ac00-4e21-aee6-b769b50f5a87-proxy-tls\") pod \"68a8729c-ac00-4e21-aee6-b769b50f5a87\" (UID: \"68a8729c-ac00-4e21-aee6-b769b50f5a87\") "
Jul 07 15:14:13.356063 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.356033    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68a8729c-ac00-4e21-aee6-b769b50f5a87-kube-rbac-proxy-sar-config\") pod \"68a8729c-ac00-4e21-aee6-b769b50f5a87\" (UID: \"68a8729c-ac00-4e21-aee6-b769b50f5a87\") "
Jul 07 15:14:13.356471 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.356446    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a8729c-ac00-4e21-aee6-b769b50f5a87-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "68a8729c-ac00-4e21-aee6-b769b50f5a87" (UID: "68a8729c-ac00-4e21-aee6-b769b50f5a87"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:14:13.357940 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.357921    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68a8729c-ac00-4e21-aee6-b769b50f5a87-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "68a8729c-ac00-4e21-aee6-b769b50f5a87" (UID: "68a8729c-ac00-4e21-aee6-b769b50f5a87"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:14:13.357940 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.357920    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a8729c-ac00-4e21-aee6-b769b50f5a87-kube-api-access-wshqt" (OuterVolumeSpecName: "kube-api-access-wshqt") pod "68a8729c-ac00-4e21-aee6-b769b50f5a87" (UID: "68a8729c-ac00-4e21-aee6-b769b50f5a87"). InnerVolumeSpecName "kube-api-access-wshqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:14:13.358070 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.357995    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:13.378492 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.378465    2581 generic.go:356] "Generic (PLEG): container finished" podID="066b718a-05bd-447a-b270-d84635d62900" containerID="a06c8da10e492b1d005cb8970a897719d913f68d83bee15d057c9b568f287d8b" exitCode=2
Jul 07 15:14:13.378631 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.378543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" event={"ID":"066b718a-05bd-447a-b270-d84635d62900","Type":"ContainerDied","Data":"a06c8da10e492b1d005cb8970a897719d913f68d83bee15d057c9b568f287d8b"}
Jul 07 15:14:13.380250 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.380225    2581 generic.go:356] "Generic (PLEG): container finished" podID="68a8729c-ac00-4e21-aee6-b769b50f5a87" containerID="07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a" exitCode=2
Jul 07 15:14:13.380250 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.380243    2581 generic.go:356] "Generic (PLEG): container finished" podID="68a8729c-ac00-4e21-aee6-b769b50f5a87" containerID="f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0" exitCode=2
Jul 07 15:14:13.380486 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.380261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr" event={"ID":"68a8729c-ac00-4e21-aee6-b769b50f5a87","Type":"ContainerDied","Data":"07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a"}
Jul 07 15:14:13.380486 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.380281    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"
Jul 07 15:14:13.380486 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.380304    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr" event={"ID":"68a8729c-ac00-4e21-aee6-b769b50f5a87","Type":"ContainerDied","Data":"f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0"}
Jul 07 15:14:13.380486 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.380324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr" event={"ID":"68a8729c-ac00-4e21-aee6-b769b50f5a87","Type":"ContainerDied","Data":"08212a666aa44fc7c950570fb47d95ae0898da55063884b8958d53d8e060275a"}
Jul 07 15:14:13.380486 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.380362    2581 scope.go:117] "RemoveContainer" containerID="07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a"
Jul 07 15:14:13.389985 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.389967    2581 scope.go:117] "RemoveContainer" containerID="f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0"
Jul 07 15:14:13.399252 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.399233    2581 scope.go:117] "RemoveContainer" containerID="07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a"
Jul 07 15:14:13.400157 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:14:13.399595    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a\": container with ID starting with 07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a not found: ID does not exist" containerID="07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a"
Jul 07 15:14:13.400157 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.399632    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a"} err="failed to get container status \"07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a\": rpc error: code = NotFound desc = could not find container \"07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a\": container with ID starting with 07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a not found: ID does not exist"
Jul 07 15:14:13.400157 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.399656    2581 scope.go:117] "RemoveContainer" containerID="f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0"
Jul 07 15:14:13.400157 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:14:13.399905    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0\": container with ID starting with f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0 not found: ID does not exist" containerID="f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0"
Jul 07 15:14:13.400157 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.399929    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0"} err="failed to get container status \"f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0\": rpc error: code = NotFound desc = could not find container \"f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0\": container with ID starting with f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0 not found: ID does not exist"
Jul 07 15:14:13.400157 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.399949    2581 scope.go:117] "RemoveContainer" containerID="07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a"
Jul 07 15:14:13.400525 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.400166    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a"} err="failed to get container status \"07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a\": rpc error: code = NotFound desc = could not find container \"07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a\": container with ID starting with 07a1b24837fd6655c440f7966f0b3656df72932504b7e520ed8a0546c1cf190a not found: ID does not exist"
Jul 07 15:14:13.400525 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.400188    2581 scope.go:117] "RemoveContainer" containerID="f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0"
Jul 07 15:14:13.400525 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.400427    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0"} err="failed to get container status \"f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0\": rpc error: code = NotFound desc = could not find container \"f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0\": container with ID starting with f80e98319b495bf99196401f644c6c929c81abde3b064f9109c5c05d1585ace0 not found: ID does not exist"
Jul 07 15:14:13.421302 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.421216    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"]
Jul 07 15:14:13.428477 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.428448    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c27ac-predictor-66f488f4f4-zvdmr"]
Jul 07 15:14:13.456979 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.456948    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wshqt\" (UniqueName: \"kubernetes.io/projected/68a8729c-ac00-4e21-aee6-b769b50f5a87-kube-api-access-wshqt\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:14:13.456979 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.456973    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68a8729c-ac00-4e21-aee6-b769b50f5a87-proxy-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:14:13.457163 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.456989    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68a8729c-ac00-4e21-aee6-b769b50f5a87-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:14:13.497912 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:13.497884    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"]
Jul 07 15:14:13.500336 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:14:13.500308    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d74119_b14c_49a4_b638_275bf78f6536.slice/crio-8214a47fef79e1066a35b1b940eab4d0fff24032deb8c6c1a1f80ece0b605cd1 WatchSource:0}: Error finding container 8214a47fef79e1066a35b1b940eab4d0fff24032deb8c6c1a1f80ece0b605cd1: Status 404 returned error can't find the container with id 8214a47fef79e1066a35b1b940eab4d0fff24032deb8c6c1a1f80ece0b605cd1
Jul 07 15:14:14.386157 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:14.386061    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" event={"ID":"75d74119-b14c-49a4-b638-275bf78f6536","Type":"ContainerStarted","Data":"4bf8623bfadb1ed744aed8dc45762d9dd1fab47455a05869e3c98105d4ef6055"}
Jul 07 15:14:14.386157 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:14.386127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" event={"ID":"75d74119-b14c-49a4-b638-275bf78f6536","Type":"ContainerStarted","Data":"8214a47fef79e1066a35b1b940eab4d0fff24032deb8c6c1a1f80ece0b605cd1"}
Jul 07 15:14:14.932602 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:14.932571    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68a8729c-ac00-4e21-aee6-b769b50f5a87" path="/var/lib/kubelet/pods/68a8729c-ac00-4e21-aee6-b769b50f5a87/volumes"
Jul 07 15:14:15.991369 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:15.991315    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 07 15:14:18.406638 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:18.406602    2581 generic.go:356] "Generic (PLEG): container finished" podID="75d74119-b14c-49a4-b638-275bf78f6536" containerID="4bf8623bfadb1ed744aed8dc45762d9dd1fab47455a05869e3c98105d4ef6055" exitCode=0
Jul 07 15:14:18.407026 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:18.406677    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" event={"ID":"75d74119-b14c-49a4-b638-275bf78f6536","Type":"ContainerDied","Data":"4bf8623bfadb1ed744aed8dc45762d9dd1fab47455a05869e3c98105d4ef6055"}
Jul 07 15:14:18.409399 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:18.409373    2581 generic.go:356] "Generic (PLEG): container finished" podID="066b718a-05bd-447a-b270-d84635d62900" containerID="ff632fd676930d0eac480283aa782c48c65bb4d62c8bb72a63730d7de2ec2822" exitCode=0
Jul 07 15:14:18.409465 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:18.409436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" event={"ID":"066b718a-05bd-447a-b270-d84635d62900","Type":"ContainerDied","Data":"ff632fd676930d0eac480283aa782c48c65bb4d62c8bb72a63730d7de2ec2822"}
Jul 07 15:14:19.416219 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:19.416183    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" event={"ID":"75d74119-b14c-49a4-b638-275bf78f6536","Type":"ContainerStarted","Data":"8b21efab61e0682ec26ee06c2b62ee4b4c560c9861e685083c8c0be04a394729"}
Jul 07 15:14:19.416671 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:19.416245    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" event={"ID":"75d74119-b14c-49a4-b638-275bf78f6536","Type":"ContainerStarted","Data":"999804d18b9ec4bdfd24df881830891c6ec38b84b884b06a3bf565a214cce5eb"}
Jul 07 15:14:19.439705 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:19.439647    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podStartSLOduration=2.5175078920000002 podStartE2EDuration="6.439629207s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.922121315s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:19.438103739 +0000 UTC m=+1021.170716074" watchObservedRunningTime="2026-07-07 15:14:19.439629207 +0000 UTC m=+1021.172241526"
Jul 07 15:14:20.991076 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:20.991030    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 07 15:14:20.995449 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:20.995419    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:14:20.995786 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:20.995757    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:14:24.416868 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:24.416832    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:24.417310 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:24.417239    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:24.418207 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:24.418179    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:14:24.422957 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:24.422934    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:14:24.436757 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:24.436727    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:14:25.441554 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:25.441511    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:14:25.990804 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:25.990763    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 07 15:14:25.990979 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:25.990905    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:14:30.991372 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:30.991323    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 07 15:14:30.995649 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:30.995619    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:14:30.995974 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:30.995950    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:14:35.441965 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:35.441917    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:14:35.991159 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:35.991108    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 07 15:14:40.991145 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:40.991103    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 07 15:14:40.995480 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:40.995442    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:14:40.995614 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:40.995584    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:14:40.995781 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:40.995752    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:14:40.995875 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:40.995862    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:14:43.183969 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.183940    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:14:43.323459 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.323377    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zmpg\" (UniqueName: \"kubernetes.io/projected/066b718a-05bd-447a-b270-d84635d62900-kube-api-access-4zmpg\") pod \"066b718a-05bd-447a-b270-d84635d62900\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") "
Jul 07 15:14:43.323459 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.323436    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066b718a-05bd-447a-b270-d84635d62900-proxy-tls\") pod \"066b718a-05bd-447a-b270-d84635d62900\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") "
Jul 07 15:14:43.323459 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.323460    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066b718a-05bd-447a-b270-d84635d62900-cabundle-cert\") pod \"066b718a-05bd-447a-b270-d84635d62900\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") "
Jul 07 15:14:43.323697 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.323519    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066b718a-05bd-447a-b270-d84635d62900-kserve-provision-location\") pod \"066b718a-05bd-447a-b270-d84635d62900\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") "
Jul 07 15:14:43.323753 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.323718    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/066b718a-05bd-447a-b270-d84635d62900-kube-rbac-proxy-sar-config\") pod \"066b718a-05bd-447a-b270-d84635d62900\" (UID: \"066b718a-05bd-447a-b270-d84635d62900\") "
Jul 07 15:14:43.323833 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.323809    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066b718a-05bd-447a-b270-d84635d62900-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "066b718a-05bd-447a-b270-d84635d62900" (UID: "066b718a-05bd-447a-b270-d84635d62900"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:14:43.323833 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.323817    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066b718a-05bd-447a-b270-d84635d62900-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "066b718a-05bd-447a-b270-d84635d62900" (UID: "066b718a-05bd-447a-b270-d84635d62900"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:14:43.324032 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.324013    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066b718a-05bd-447a-b270-d84635d62900-cabundle-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:14:43.324085 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.324038    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066b718a-05bd-447a-b270-d84635d62900-kserve-provision-location\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:14:43.324179 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.324150    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066b718a-05bd-447a-b270-d84635d62900-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "066b718a-05bd-447a-b270-d84635d62900" (UID: "066b718a-05bd-447a-b270-d84635d62900"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:14:43.325671 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.325649    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/066b718a-05bd-447a-b270-d84635d62900-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "066b718a-05bd-447a-b270-d84635d62900" (UID: "066b718a-05bd-447a-b270-d84635d62900"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:14:43.325842 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.325820    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066b718a-05bd-447a-b270-d84635d62900-kube-api-access-4zmpg" (OuterVolumeSpecName: "kube-api-access-4zmpg") pod "066b718a-05bd-447a-b270-d84635d62900" (UID: "066b718a-05bd-447a-b270-d84635d62900"). InnerVolumeSpecName "kube-api-access-4zmpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:14:43.424804 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.424765    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zmpg\" (UniqueName: \"kubernetes.io/projected/066b718a-05bd-447a-b270-d84635d62900-kube-api-access-4zmpg\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:14:43.424804 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.424798    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066b718a-05bd-447a-b270-d84635d62900-proxy-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:14:43.424804 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.424809    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/066b718a-05bd-447a-b270-d84635d62900-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:14:43.521865 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.521825    2581 generic.go:356] "Generic (PLEG): container finished" podID="066b718a-05bd-447a-b270-d84635d62900" containerID="086f06834d25361ede60de872e3907746aea383f26ae5140d82b2c0281fd7d3c" exitCode=0
Jul 07 15:14:43.522028 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.521882    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"
Jul 07 15:14:43.522028 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.521904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" event={"ID":"066b718a-05bd-447a-b270-d84635d62900","Type":"ContainerDied","Data":"086f06834d25361ede60de872e3907746aea383f26ae5140d82b2c0281fd7d3c"}
Jul 07 15:14:43.522028 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.521972    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9" event={"ID":"066b718a-05bd-447a-b270-d84635d62900","Type":"ContainerDied","Data":"61253b3e577bf7728d710576eeb9ce06c1cc8c0e5ec7854232ec09b9ce175ccd"}
Jul 07 15:14:43.522028 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.522006    2581 scope.go:117] "RemoveContainer" containerID="086f06834d25361ede60de872e3907746aea383f26ae5140d82b2c0281fd7d3c"
Jul 07 15:14:43.531206 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.531188    2581 scope.go:117] "RemoveContainer" containerID="a06c8da10e492b1d005cb8970a897719d913f68d83bee15d057c9b568f287d8b"
Jul 07 15:14:43.538619 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.538600    2581 scope.go:117] "RemoveContainer" containerID="ff632fd676930d0eac480283aa782c48c65bb4d62c8bb72a63730d7de2ec2822"
Jul 07 15:14:43.546735 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.546717    2581 scope.go:117] "RemoveContainer" containerID="58c4d29783feb2383c1948538a6c1f507afb47c1586770944724797ef20dccd7"
Jul 07 15:14:43.547776 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.547710    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"]
Jul 07 15:14:43.551112 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.551091    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c27ac-predictor-54d7f5d474-l7ml9"]
Jul 07 15:14:43.554802 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.554783    2581 scope.go:117] "RemoveContainer" containerID="086f06834d25361ede60de872e3907746aea383f26ae5140d82b2c0281fd7d3c"
Jul 07 15:14:43.555101 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:14:43.555083    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"086f06834d25361ede60de872e3907746aea383f26ae5140d82b2c0281fd7d3c\": container with ID starting with 086f06834d25361ede60de872e3907746aea383f26ae5140d82b2c0281fd7d3c not found: ID does not exist" containerID="086f06834d25361ede60de872e3907746aea383f26ae5140d82b2c0281fd7d3c"
Jul 07 15:14:43.555150 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.555112    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086f06834d25361ede60de872e3907746aea383f26ae5140d82b2c0281fd7d3c"} err="failed to get container status \"086f06834d25361ede60de872e3907746aea383f26ae5140d82b2c0281fd7d3c\": rpc error: code = NotFound desc = could not find container \"086f06834d25361ede60de872e3907746aea383f26ae5140d82b2c0281fd7d3c\": container with ID starting with 086f06834d25361ede60de872e3907746aea383f26ae5140d82b2c0281fd7d3c not found: ID does not exist"
Jul 07 15:14:43.555150 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.555132    2581 scope.go:117] "RemoveContainer" containerID="a06c8da10e492b1d005cb8970a897719d913f68d83bee15d057c9b568f287d8b"
Jul 07 15:14:43.555427 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:14:43.555398    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06c8da10e492b1d005cb8970a897719d913f68d83bee15d057c9b568f287d8b\": container with ID starting with a06c8da10e492b1d005cb8970a897719d913f68d83bee15d057c9b568f287d8b not found: ID does not exist" containerID="a06c8da10e492b1d005cb8970a897719d913f68d83bee15d057c9b568f287d8b"
Jul 07 15:14:43.555500 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.555427    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06c8da10e492b1d005cb8970a897719d913f68d83bee15d057c9b568f287d8b"} err="failed to get container status \"a06c8da10e492b1d005cb8970a897719d913f68d83bee15d057c9b568f287d8b\": rpc error: code = NotFound desc = could not find container \"a06c8da10e492b1d005cb8970a897719d913f68d83bee15d057c9b568f287d8b\": container with ID starting with a06c8da10e492b1d005cb8970a897719d913f68d83bee15d057c9b568f287d8b not found: ID does not exist"
Jul 07 15:14:43.555500 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.555443    2581 scope.go:117] "RemoveContainer" containerID="ff632fd676930d0eac480283aa782c48c65bb4d62c8bb72a63730d7de2ec2822"
Jul 07 15:14:43.555691 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:14:43.555671    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff632fd676930d0eac480283aa782c48c65bb4d62c8bb72a63730d7de2ec2822\": container with ID starting with ff632fd676930d0eac480283aa782c48c65bb4d62c8bb72a63730d7de2ec2822 not found: ID does not exist" containerID="ff632fd676930d0eac480283aa782c48c65bb4d62c8bb72a63730d7de2ec2822"
Jul 07 15:14:43.555732 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.555698    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff632fd676930d0eac480283aa782c48c65bb4d62c8bb72a63730d7de2ec2822"} err="failed to get container status \"ff632fd676930d0eac480283aa782c48c65bb4d62c8bb72a63730d7de2ec2822\": rpc error: code = NotFound desc = could not find container \"ff632fd676930d0eac480283aa782c48c65bb4d62c8bb72a63730d7de2ec2822\": container with ID starting with ff632fd676930d0eac480283aa782c48c65bb4d62c8bb72a63730d7de2ec2822 not found: ID does not exist"
Jul 07 15:14:43.555732 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.555714    2581 scope.go:117] "RemoveContainer" containerID="58c4d29783feb2383c1948538a6c1f507afb47c1586770944724797ef20dccd7"
Jul 07 15:14:43.555976 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:14:43.555958    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c4d29783feb2383c1948538a6c1f507afb47c1586770944724797ef20dccd7\": container with ID starting with 58c4d29783feb2383c1948538a6c1f507afb47c1586770944724797ef20dccd7 not found: ID does not exist" containerID="58c4d29783feb2383c1948538a6c1f507afb47c1586770944724797ef20dccd7"
Jul 07 15:14:43.556018 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:43.555981    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c4d29783feb2383c1948538a6c1f507afb47c1586770944724797ef20dccd7"} err="failed to get container status \"58c4d29783feb2383c1948538a6c1f507afb47c1586770944724797ef20dccd7\": rpc error: code = NotFound desc = could not find container \"58c4d29783feb2383c1948538a6c1f507afb47c1586770944724797ef20dccd7\": container with ID starting with 58c4d29783feb2383c1948538a6c1f507afb47c1586770944724797ef20dccd7 not found: ID does not exist"
Jul 07 15:14:44.932865 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:44.932832    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="066b718a-05bd-447a-b270-d84635d62900" path="/var/lib/kubelet/pods/066b718a-05bd-447a-b270-d84635d62900/volumes"
Jul 07 15:14:45.441729 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:45.441693    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:14:55.442451 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:14:55.442410    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:15:05.441498 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:15:05.441455    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:15:15.442369 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:15:15.442297    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:15:25.442164 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:15:25.442117    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:15:27.928057 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:15:27.928008    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:15:37.928494 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:15:37.928449    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:15:47.927963 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:15:47.927922    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:15:57.928260 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:15:57.928211    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:16:07.927852 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:16:07.927811    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:16:17.928695 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:16:17.928643    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:16:27.928547 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:16:27.928500    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:16:37.928649 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:16:37.928592    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:16:41.927855 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:16:41.927761    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:16:51.928720 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:16:51.928675    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:17:01.929311 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:01.929273    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:17:03.166932 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:03.166896    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"]
Jul 07 15:17:03.167420 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:03.167222    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kube-rbac-proxy" containerID="cri-o://8b21efab61e0682ec26ee06c2b62ee4b4c560c9861e685083c8c0be04a394729" gracePeriod=30
Jul 07 15:17:03.167420 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:03.167219    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" containerID="cri-o://999804d18b9ec4bdfd24df881830891c6ec38b84b884b06a3bf565a214cce5eb" gracePeriod=30
Jul 07 15:17:04.136097 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:04.136065    2581 generic.go:356] "Generic (PLEG): container finished" podID="75d74119-b14c-49a4-b638-275bf78f6536" containerID="8b21efab61e0682ec26ee06c2b62ee4b4c560c9861e685083c8c0be04a394729" exitCode=2
Jul 07 15:17:04.136275 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:04.136117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" event={"ID":"75d74119-b14c-49a4-b638-275bf78f6536","Type":"ContainerDied","Data":"8b21efab61e0682ec26ee06c2b62ee4b4c560c9861e685083c8c0be04a394729"}
Jul 07 15:17:04.424061 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:04.424020    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 07 15:17:08.310226 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.310186    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"]
Jul 07 15:17:08.311561 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.311533    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="storage-initializer"
Jul 07 15:17:08.311770 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.311744    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="storage-initializer"
Jul 07 15:17:08.311836 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.311783    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kube-rbac-proxy"
Jul 07 15:17:08.311836 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.311795    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kube-rbac-proxy"
Jul 07 15:17:08.311836 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.311829    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68a8729c-ac00-4e21-aee6-b769b50f5a87" containerName="kube-rbac-proxy"
Jul 07 15:17:08.311836 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.311837    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="68a8729c-ac00-4e21-aee6-b769b50f5a87" containerName="kube-rbac-proxy"
Jul 07 15:17:08.312020 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.311847    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container"
Jul 07 15:17:08.312020 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.311856    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container"
Jul 07 15:17:08.312020 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.311875    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68a8729c-ac00-4e21-aee6-b769b50f5a87" containerName="kserve-container"
Jul 07 15:17:08.312020 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.311883    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="68a8729c-ac00-4e21-aee6-b769b50f5a87" containerName="kserve-container"
Jul 07 15:17:08.312020 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.311893    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent"
Jul 07 15:17:08.312020 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.311901    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent"
Jul 07 15:17:08.312282 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.312110    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="68a8729c-ac00-4e21-aee6-b769b50f5a87" containerName="kserve-container"
Jul 07 15:17:08.312282 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.312124    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kserve-container"
Jul 07 15:17:08.312282 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.312133    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="kube-rbac-proxy"
Jul 07 15:17:08.312282 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.312147    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="066b718a-05bd-447a-b270-d84635d62900" containerName="agent"
Jul 07 15:17:08.312282 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.312156    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="68a8729c-ac00-4e21-aee6-b769b50f5a87" containerName="kube-rbac-proxy"
Jul 07 15:17:08.316670 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.316648    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.319712 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.319687    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d3e813-predictor-serving-cert\""
Jul 07 15:17:08.319893 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.319765    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d3e813-kube-rbac-proxy-sar-config\""
Jul 07 15:17:08.322439 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.322416    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"]
Jul 07 15:17:08.344592 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.344555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-proxy-tls\") pod \"isvc-primary-d3e813-predictor-5ccd495944-jplx5\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") " pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.344734 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.344618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh4sv\" (UniqueName: \"kubernetes.io/projected/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kube-api-access-dh4sv\") pod \"isvc-primary-d3e813-predictor-5ccd495944-jplx5\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") " pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.344734 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.344656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d3e813-predictor-5ccd495944-jplx5\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") " pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.344734 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.344684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kserve-provision-location\") pod \"isvc-primary-d3e813-predictor-5ccd495944-jplx5\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") " pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.344848 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.344733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-cabundle-cert\") pod \"isvc-primary-d3e813-predictor-5ccd495944-jplx5\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") " pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.445810 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.445773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-proxy-tls\") pod \"isvc-primary-d3e813-predictor-5ccd495944-jplx5\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") " pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.446006 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.445927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh4sv\" (UniqueName: \"kubernetes.io/projected/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kube-api-access-dh4sv\") pod \"isvc-primary-d3e813-predictor-5ccd495944-jplx5\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") " pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.446006 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.445973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d3e813-predictor-5ccd495944-jplx5\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") " pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.446127 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.446040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kserve-provision-location\") pod \"isvc-primary-d3e813-predictor-5ccd495944-jplx5\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") " pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.446127 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.446093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-cabundle-cert\") pod \"isvc-primary-d3e813-predictor-5ccd495944-jplx5\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") " pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.446524 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.446496    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kserve-provision-location\") pod \"isvc-primary-d3e813-predictor-5ccd495944-jplx5\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") " pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.446663 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.446642    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d3e813-predictor-5ccd495944-jplx5\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") " pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.446764 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.446747    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-cabundle-cert\") pod \"isvc-primary-d3e813-predictor-5ccd495944-jplx5\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") " pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.448185 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.448166    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-proxy-tls\") pod \"isvc-primary-d3e813-predictor-5ccd495944-jplx5\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") " pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.456265 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.456234    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh4sv\" (UniqueName: \"kubernetes.io/projected/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kube-api-access-dh4sv\") pod \"isvc-primary-d3e813-predictor-5ccd495944-jplx5\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") " pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.628444 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.628340    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:08.762487 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:08.762461    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"]
Jul 07 15:17:08.765182 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:17:08.765156    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49682b8_c9c8_4d9f_a25d_b93afb1701ff.slice/crio-b22ab17e6427bab27fbb78cef54badd779be118f47dec60ffdac993d1d7487cd WatchSource:0}: Error finding container b22ab17e6427bab27fbb78cef54badd779be118f47dec60ffdac993d1d7487cd: Status 404 returned error can't find the container with id b22ab17e6427bab27fbb78cef54badd779be118f47dec60ffdac993d1d7487cd
Jul 07 15:17:09.158196 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:09.158161    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" event={"ID":"a49682b8-c9c8-4d9f-a25d-b93afb1701ff","Type":"ContainerStarted","Data":"ea2a588e19f36380c6bb0fa1939d7f5f0effca0cb1b0375f097d0d03acbaf54d"}
Jul 07 15:17:09.158390 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:09.158228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" event={"ID":"a49682b8-c9c8-4d9f-a25d-b93afb1701ff","Type":"ContainerStarted","Data":"b22ab17e6427bab27fbb78cef54badd779be118f47dec60ffdac993d1d7487cd"}
Jul 07 15:17:09.424221 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:09.424187    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 07 15:17:11.927888 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:11.927841    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:17:13.176806 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.176772    2581 generic.go:356] "Generic (PLEG): container finished" podID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerID="ea2a588e19f36380c6bb0fa1939d7f5f0effca0cb1b0375f097d0d03acbaf54d" exitCode=0
Jul 07 15:17:13.177214 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.176852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" event={"ID":"a49682b8-c9c8-4d9f-a25d-b93afb1701ff","Type":"ContainerDied","Data":"ea2a588e19f36380c6bb0fa1939d7f5f0effca0cb1b0375f097d0d03acbaf54d"}
Jul 07 15:17:13.179282 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.179260    2581 generic.go:356] "Generic (PLEG): container finished" podID="75d74119-b14c-49a4-b638-275bf78f6536" containerID="999804d18b9ec4bdfd24df881830891c6ec38b84b884b06a3bf565a214cce5eb" exitCode=0
Jul 07 15:17:13.179413 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.179343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" event={"ID":"75d74119-b14c-49a4-b638-275bf78f6536","Type":"ContainerDied","Data":"999804d18b9ec4bdfd24df881830891c6ec38b84b884b06a3bf565a214cce5eb"}
Jul 07 15:17:13.232039 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.232015    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:17:13.286284 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.286255    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj6tj\" (UniqueName: \"kubernetes.io/projected/75d74119-b14c-49a4-b638-275bf78f6536-kube-api-access-gj6tj\") pod \"75d74119-b14c-49a4-b638-275bf78f6536\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") "
Jul 07 15:17:13.286431 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.286300    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75d74119-b14c-49a4-b638-275bf78f6536-kserve-provision-location\") pod \"75d74119-b14c-49a4-b638-275bf78f6536\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") "
Jul 07 15:17:13.286431 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.286328    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75d74119-b14c-49a4-b638-275bf78f6536-proxy-tls\") pod \"75d74119-b14c-49a4-b638-275bf78f6536\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") "
Jul 07 15:17:13.286431 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.286368    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75d74119-b14c-49a4-b638-275bf78f6536-kube-rbac-proxy-sar-config\") pod \"75d74119-b14c-49a4-b638-275bf78f6536\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") "
Jul 07 15:17:13.286431 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.286431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75d74119-b14c-49a4-b638-275bf78f6536-cabundle-cert\") pod \"75d74119-b14c-49a4-b638-275bf78f6536\" (UID: \"75d74119-b14c-49a4-b638-275bf78f6536\") "
Jul 07 15:17:13.286701 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.286658    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d74119-b14c-49a4-b638-275bf78f6536-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75d74119-b14c-49a4-b638-275bf78f6536" (UID: "75d74119-b14c-49a4-b638-275bf78f6536"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:17:13.286829 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.286755    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75d74119-b14c-49a4-b638-275bf78f6536-kserve-provision-location\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:17:13.286829 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.286755    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d74119-b14c-49a4-b638-275bf78f6536-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "75d74119-b14c-49a4-b638-275bf78f6536" (UID: "75d74119-b14c-49a4-b638-275bf78f6536"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:17:13.287109 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.287089    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d74119-b14c-49a4-b638-275bf78f6536-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75d74119-b14c-49a4-b638-275bf78f6536" (UID: "75d74119-b14c-49a4-b638-275bf78f6536"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:17:13.288741 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.288719    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d74119-b14c-49a4-b638-275bf78f6536-kube-api-access-gj6tj" (OuterVolumeSpecName: "kube-api-access-gj6tj") pod "75d74119-b14c-49a4-b638-275bf78f6536" (UID: "75d74119-b14c-49a4-b638-275bf78f6536"). InnerVolumeSpecName "kube-api-access-gj6tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:17:13.288830 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.288786    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d74119-b14c-49a4-b638-275bf78f6536-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75d74119-b14c-49a4-b638-275bf78f6536" (UID: "75d74119-b14c-49a4-b638-275bf78f6536"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:17:13.387931 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.387896    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75d74119-b14c-49a4-b638-275bf78f6536-proxy-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:17:13.387931 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.387925    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75d74119-b14c-49a4-b638-275bf78f6536-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:17:13.387931 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.387935    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75d74119-b14c-49a4-b638-275bf78f6536-cabundle-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:17:13.387931 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:13.387943    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj6tj\" (UniqueName: \"kubernetes.io/projected/75d74119-b14c-49a4-b638-275bf78f6536-kube-api-access-gj6tj\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:17:14.185904 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:14.185868    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" event={"ID":"a49682b8-c9c8-4d9f-a25d-b93afb1701ff","Type":"ContainerStarted","Data":"b00e8b7432246bcb76373e9461d726071eabab47c1a1d8d93d8064d4543d20de"}
Jul 07 15:17:14.186423 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:14.185938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" event={"ID":"a49682b8-c9c8-4d9f-a25d-b93afb1701ff","Type":"ContainerStarted","Data":"1c9cf60fec0cad1877db590e6d926435c7d6f8b90addcd951dcad4eebd772439"}
Jul 07 15:17:14.186423 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:14.186166    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:14.187843 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:14.187825    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"
Jul 07 15:17:14.187976 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:14.187843    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx" event={"ID":"75d74119-b14c-49a4-b638-275bf78f6536","Type":"ContainerDied","Data":"8214a47fef79e1066a35b1b940eab4d0fff24032deb8c6c1a1f80ece0b605cd1"}
Jul 07 15:17:14.187976 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:14.187894    2581 scope.go:117] "RemoveContainer" containerID="8b21efab61e0682ec26ee06c2b62ee4b4c560c9861e685083c8c0be04a394729"
Jul 07 15:17:14.196757 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:14.196726    2581 scope.go:117] "RemoveContainer" containerID="999804d18b9ec4bdfd24df881830891c6ec38b84b884b06a3bf565a214cce5eb"
Jul 07 15:17:14.204677 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:14.204659    2581 scope.go:117] "RemoveContainer" containerID="4bf8623bfadb1ed744aed8dc45762d9dd1fab47455a05869e3c98105d4ef6055"
Jul 07 15:17:14.208456 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:14.208405    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" podStartSLOduration=2.560158553 podStartE2EDuration="6.208332417s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.648173864s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:17:14.207006005 +0000 UTC m=+1195.939618340" watchObservedRunningTime="2026-07-07 15:17:14.208332417 +0000 UTC m=+1195.940944736"
Jul 07 15:17:14.223456 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:14.223433    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"]
Jul 07 15:17:14.227219 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:14.227192    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-11363-predictor-8dccf66dc-lkbcx"]
Jul 07 15:17:14.932237 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:14.932202    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75d74119-b14c-49a4-b638-275bf78f6536" path="/var/lib/kubelet/pods/75d74119-b14c-49a4-b638-275bf78f6536/volumes"
Jul 07 15:17:15.193651 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:15.193558    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:15.194765 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:15.194742    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:17:16.197096 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:16.197051    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:17:21.202150 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:21.202119    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:17:21.207903 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:21.207841    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:17:31.202778 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:31.202737    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:17:41.203212 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:41.203173    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:17:51.203043 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:17:51.202997    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:18:01.203207 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:01.203166    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:18:11.203249 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:11.203166    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:18:21.203435 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:21.203388    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:18:29.927564 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:29.927523    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:18:38.566913 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.566877    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"]
Jul 07 15:18:38.567298 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.567274    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="storage-initializer"
Jul 07 15:18:38.567298 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.567288    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="storage-initializer"
Jul 07 15:18:38.567410 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.567305    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container"
Jul 07 15:18:38.567410 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.567312    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container"
Jul 07 15:18:38.567410 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.567329    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kube-rbac-proxy"
Jul 07 15:18:38.567410 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.567337    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kube-rbac-proxy"
Jul 07 15:18:38.567585 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.567471    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kserve-container"
Jul 07 15:18:38.567585 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.567484    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="75d74119-b14c-49a4-b638-275bf78f6536" containerName="kube-rbac-proxy"
Jul 07 15:18:38.570616 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.570600    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:38.573550 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.573535    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d3e813-predictor-serving-cert\""
Jul 07 15:18:38.573647 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.573540    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-d3e813\""
Jul 07 15:18:38.573883 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.573864    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d3e813-kube-rbac-proxy-sar-config\""
Jul 07 15:18:38.573964 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.573911    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-d3e813-dockercfg-4xkkw\""
Jul 07 15:18:38.581389 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.581365    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"]
Jul 07 15:18:38.750055 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.749995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a27b72f-4351-45a7-8978-2770feeb7d7f-proxy-tls\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:38.750055 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.750048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a27b72f-4351-45a7-8978-2770feeb7d7f-kserve-provision-location\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:38.750272 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.750084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7dxj\" (UniqueName: \"kubernetes.io/projected/6a27b72f-4351-45a7-8978-2770feeb7d7f-kube-api-access-w7dxj\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:38.750272 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.750212    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a27b72f-4351-45a7-8978-2770feeb7d7f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:38.750272 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.750232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a27b72f-4351-45a7-8978-2770feeb7d7f-cabundle-cert\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:38.851545 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.851458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a27b72f-4351-45a7-8978-2770feeb7d7f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:38.851545 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.851498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a27b72f-4351-45a7-8978-2770feeb7d7f-cabundle-cert\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:38.851545 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.851522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a27b72f-4351-45a7-8978-2770feeb7d7f-proxy-tls\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:38.851545 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.851538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a27b72f-4351-45a7-8978-2770feeb7d7f-kserve-provision-location\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:38.851859 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.851558    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7dxj\" (UniqueName: \"kubernetes.io/projected/6a27b72f-4351-45a7-8978-2770feeb7d7f-kube-api-access-w7dxj\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:38.851859 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:18:38.851669    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-serving-cert: secret "isvc-secondary-d3e813-predictor-serving-cert" not found
Jul 07 15:18:38.851859 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:18:38.851766    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a27b72f-4351-45a7-8978-2770feeb7d7f-proxy-tls podName:6a27b72f-4351-45a7-8978-2770feeb7d7f nodeName:}" failed. No retries permitted until 2026-07-07 15:18:39.351741679 +0000 UTC m=+1281.084353999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6a27b72f-4351-45a7-8978-2770feeb7d7f-proxy-tls") pod "isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8" (UID: "6a27b72f-4351-45a7-8978-2770feeb7d7f") : secret "isvc-secondary-d3e813-predictor-serving-cert" not found
Jul 07 15:18:38.852025 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.851992    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a27b72f-4351-45a7-8978-2770feeb7d7f-kserve-provision-location\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:38.852246 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.852227    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a27b72f-4351-45a7-8978-2770feeb7d7f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:38.852283 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.852228    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a27b72f-4351-45a7-8978-2770feeb7d7f-cabundle-cert\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:38.862375 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:38.862341    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7dxj\" (UniqueName: \"kubernetes.io/projected/6a27b72f-4351-45a7-8978-2770feeb7d7f-kube-api-access-w7dxj\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:39.356622 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:39.356581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a27b72f-4351-45a7-8978-2770feeb7d7f-proxy-tls\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:39.358954 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:39.358924    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a27b72f-4351-45a7-8978-2770feeb7d7f-proxy-tls\") pod \"isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") " pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:39.481530 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:39.481466    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:39.623905 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:39.623880    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"]
Jul 07 15:18:39.625735 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:18:39.625710    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a27b72f_4351_45a7_8978_2770feeb7d7f.slice/crio-253eb47fd0436e8366d7a5ad1c3d3afef299f26a3f2385ae3dc5e1e2acf8b370 WatchSource:0}: Error finding container 253eb47fd0436e8366d7a5ad1c3d3afef299f26a3f2385ae3dc5e1e2acf8b370: Status 404 returned error can't find the container with id 253eb47fd0436e8366d7a5ad1c3d3afef299f26a3f2385ae3dc5e1e2acf8b370
Jul 07 15:18:40.555155 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:40.555123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8" event={"ID":"6a27b72f-4351-45a7-8978-2770feeb7d7f","Type":"ContainerStarted","Data":"d6f3d8e712cb8f942e069a5c338bef344e739ac8eddbdae24c481fc169e7dee2"}
Jul 07 15:18:40.555335 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:40.555182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8" event={"ID":"6a27b72f-4351-45a7-8978-2770feeb7d7f","Type":"ContainerStarted","Data":"253eb47fd0436e8366d7a5ad1c3d3afef299f26a3f2385ae3dc5e1e2acf8b370"}
Jul 07 15:18:44.572890 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:44.572860    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8_6a27b72f-4351-45a7-8978-2770feeb7d7f/storage-initializer/0.log"
Jul 07 15:18:44.573285 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:44.572902    2581 generic.go:356] "Generic (PLEG): container finished" podID="6a27b72f-4351-45a7-8978-2770feeb7d7f" containerID="d6f3d8e712cb8f942e069a5c338bef344e739ac8eddbdae24c481fc169e7dee2" exitCode=1
Jul 07 15:18:44.573285 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:44.572981    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8" event={"ID":"6a27b72f-4351-45a7-8978-2770feeb7d7f","Type":"ContainerDied","Data":"d6f3d8e712cb8f942e069a5c338bef344e739ac8eddbdae24c481fc169e7dee2"}
Jul 07 15:18:45.579106 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:45.579079    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8_6a27b72f-4351-45a7-8978-2770feeb7d7f/storage-initializer/0.log"
Jul 07 15:18:45.579531 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:45.579132    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8" event={"ID":"6a27b72f-4351-45a7-8978-2770feeb7d7f","Type":"ContainerStarted","Data":"b62b3d50429c6c8d2f786f412dee239c89e5fe91de880161016c8d38d9173343"}
Jul 07 15:18:46.585030 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:46.585004    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8_6a27b72f-4351-45a7-8978-2770feeb7d7f/storage-initializer/1.log"
Jul 07 15:18:46.585463 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:46.585377    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8_6a27b72f-4351-45a7-8978-2770feeb7d7f/storage-initializer/0.log"
Jul 07 15:18:46.585463 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:46.585410    2581 generic.go:356] "Generic (PLEG): container finished" podID="6a27b72f-4351-45a7-8978-2770feeb7d7f" containerID="b62b3d50429c6c8d2f786f412dee239c89e5fe91de880161016c8d38d9173343" exitCode=1
Jul 07 15:18:46.585463 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:46.585445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8" event={"ID":"6a27b72f-4351-45a7-8978-2770feeb7d7f","Type":"ContainerDied","Data":"b62b3d50429c6c8d2f786f412dee239c89e5fe91de880161016c8d38d9173343"}
Jul 07 15:18:46.585570 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:46.585507    2581 scope.go:117] "RemoveContainer" containerID="d6f3d8e712cb8f942e069a5c338bef344e739ac8eddbdae24c481fc169e7dee2"
Jul 07 15:18:46.585868 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:46.585851    2581 scope.go:117] "RemoveContainer" containerID="d6f3d8e712cb8f942e069a5c338bef344e739ac8eddbdae24c481fc169e7dee2"
Jul 07 15:18:46.596477 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:18:46.596447    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8_kserve-ci-e2e-test(6a27b72f-4351-45a7-8978-2770feeb7d7f)\"" pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8" podUID="6a27b72f-4351-45a7-8978-2770feeb7d7f"
Jul 07 15:18:46.596604 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:18:46.596476    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8_kserve-ci-e2e-test_6a27b72f-4351-45a7-8978-2770feeb7d7f_0 in pod sandbox 253eb47fd0436e8366d7a5ad1c3d3afef299f26a3f2385ae3dc5e1e2acf8b370 from index: no such id: 'd6f3d8e712cb8f942e069a5c338bef344e739ac8eddbdae24c481fc169e7dee2'" containerID="d6f3d8e712cb8f942e069a5c338bef344e739ac8eddbdae24c481fc169e7dee2"
Jul 07 15:18:46.596604 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:46.596527    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f3d8e712cb8f942e069a5c338bef344e739ac8eddbdae24c481fc169e7dee2"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8_kserve-ci-e2e-test_6a27b72f-4351-45a7-8978-2770feeb7d7f_0 in pod sandbox 253eb47fd0436e8366d7a5ad1c3d3afef299f26a3f2385ae3dc5e1e2acf8b370 from index: no such id: 'd6f3d8e712cb8f942e069a5c338bef344e739ac8eddbdae24c481fc169e7dee2'"
Jul 07 15:18:47.591483 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:47.591457    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8_6a27b72f-4351-45a7-8978-2770feeb7d7f/storage-initializer/1.log"
Jul 07 15:18:56.549477 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.549441    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"]
Jul 07 15:18:56.647227 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.647190    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"]
Jul 07 15:18:56.647550 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.647517    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kserve-container" containerID="cri-o://1c9cf60fec0cad1877db590e6d926435c7d6f8b90addcd951dcad4eebd772439" gracePeriod=30
Jul 07 15:18:56.647697 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.647587    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kube-rbac-proxy" containerID="cri-o://b00e8b7432246bcb76373e9461d726071eabab47c1a1d8d93d8064d4543d20de" gracePeriod=30
Jul 07 15:18:56.768614 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.768592    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8_6a27b72f-4351-45a7-8978-2770feeb7d7f/storage-initializer/1.log"
Jul 07 15:18:56.770763 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.770747    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:56.809185 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.809089    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a27b72f-4351-45a7-8978-2770feeb7d7f-proxy-tls\") pod \"6a27b72f-4351-45a7-8978-2770feeb7d7f\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") "
Jul 07 15:18:56.809185 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.809140    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a27b72f-4351-45a7-8978-2770feeb7d7f-kserve-provision-location\") pod \"6a27b72f-4351-45a7-8978-2770feeb7d7f\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") "
Jul 07 15:18:56.809395 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.809249    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a27b72f-4351-45a7-8978-2770feeb7d7f-kube-rbac-proxy-sar-config\") pod \"6a27b72f-4351-45a7-8978-2770feeb7d7f\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") "
Jul 07 15:18:56.809395 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.809310    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a27b72f-4351-45a7-8978-2770feeb7d7f-cabundle-cert\") pod \"6a27b72f-4351-45a7-8978-2770feeb7d7f\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") "
Jul 07 15:18:56.809395 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.809366    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7dxj\" (UniqueName: \"kubernetes.io/projected/6a27b72f-4351-45a7-8978-2770feeb7d7f-kube-api-access-w7dxj\") pod \"6a27b72f-4351-45a7-8978-2770feeb7d7f\" (UID: \"6a27b72f-4351-45a7-8978-2770feeb7d7f\") "
Jul 07 15:18:56.809395 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.809387    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a27b72f-4351-45a7-8978-2770feeb7d7f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a27b72f-4351-45a7-8978-2770feeb7d7f" (UID: "6a27b72f-4351-45a7-8978-2770feeb7d7f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:18:56.809680 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.809648    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a27b72f-4351-45a7-8978-2770feeb7d7f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a27b72f-4351-45a7-8978-2770feeb7d7f" (UID: "6a27b72f-4351-45a7-8978-2770feeb7d7f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:18:56.809790 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.809689    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a27b72f-4351-45a7-8978-2770feeb7d7f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a27b72f-4351-45a7-8978-2770feeb7d7f" (UID: "6a27b72f-4351-45a7-8978-2770feeb7d7f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:18:56.809790 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.809710    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a27b72f-4351-45a7-8978-2770feeb7d7f-kserve-provision-location\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:18:56.809790 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.809733    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a27b72f-4351-45a7-8978-2770feeb7d7f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:18:56.811333 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.811312    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a27b72f-4351-45a7-8978-2770feeb7d7f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a27b72f-4351-45a7-8978-2770feeb7d7f" (UID: "6a27b72f-4351-45a7-8978-2770feeb7d7f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:18:56.811421 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.811308    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a27b72f-4351-45a7-8978-2770feeb7d7f-kube-api-access-w7dxj" (OuterVolumeSpecName: "kube-api-access-w7dxj") pod "6a27b72f-4351-45a7-8978-2770feeb7d7f" (UID: "6a27b72f-4351-45a7-8978-2770feeb7d7f"). InnerVolumeSpecName "kube-api-access-w7dxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:18:56.911044 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.911005    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a27b72f-4351-45a7-8978-2770feeb7d7f-proxy-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:18:56.911044 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.911037    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a27b72f-4351-45a7-8978-2770feeb7d7f-cabundle-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:18:56.911044 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:56.911046    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7dxj\" (UniqueName: \"kubernetes.io/projected/6a27b72f-4351-45a7-8978-2770feeb7d7f-kube-api-access-w7dxj\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:18:57.635541 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:57.635512    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8_6a27b72f-4351-45a7-8978-2770feeb7d7f/storage-initializer/1.log"
Jul 07 15:18:57.636023 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:57.635609    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"
Jul 07 15:18:57.636023 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:57.635615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8" event={"ID":"6a27b72f-4351-45a7-8978-2770feeb7d7f","Type":"ContainerDied","Data":"253eb47fd0436e8366d7a5ad1c3d3afef299f26a3f2385ae3dc5e1e2acf8b370"}
Jul 07 15:18:57.636023 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:57.635685    2581 scope.go:117] "RemoveContainer" containerID="b62b3d50429c6c8d2f786f412dee239c89e5fe91de880161016c8d38d9173343"
Jul 07 15:18:57.638319 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:57.638293    2581 generic.go:356] "Generic (PLEG): container finished" podID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerID="b00e8b7432246bcb76373e9461d726071eabab47c1a1d8d93d8064d4543d20de" exitCode=2
Jul 07 15:18:57.638454 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:57.638384    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" event={"ID":"a49682b8-c9c8-4d9f-a25d-b93afb1701ff","Type":"ContainerDied","Data":"b00e8b7432246bcb76373e9461d726071eabab47c1a1d8d93d8064d4543d20de"}
Jul 07 15:18:57.671715 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:57.671682    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"]
Jul 07 15:18:57.675560 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:57.675533    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d3e813-predictor-64cc95f8c-zbfw8"]
Jul 07 15:18:58.932254 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:58.932219    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a27b72f-4351-45a7-8978-2770feeb7d7f" path="/var/lib/kubelet/pods/6a27b72f-4351-45a7-8978-2770feeb7d7f/volumes"
Jul 07 15:18:59.928215 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:18:59.928174    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:19:01.198371 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.198296    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 07 15:19:01.493859 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.493833    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:19:01.551767 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.551738    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dh4sv\" (UniqueName: \"kubernetes.io/projected/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kube-api-access-dh4sv\") pod \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") "
Jul 07 15:19:01.551962 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.551794    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kube-rbac-proxy-sar-config\") pod \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") "
Jul 07 15:19:01.551962 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.551824    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-cabundle-cert\") pod \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") "
Jul 07 15:19:01.551962 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.551876    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kserve-provision-location\") pod \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") "
Jul 07 15:19:01.552122 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.551964    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-proxy-tls\") pod \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\" (UID: \"a49682b8-c9c8-4d9f-a25d-b93afb1701ff\") "
Jul 07 15:19:01.552238 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.552210    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a49682b8-c9c8-4d9f-a25d-b93afb1701ff" (UID: "a49682b8-c9c8-4d9f-a25d-b93afb1701ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:19:01.552542 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.552272    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a49682b8-c9c8-4d9f-a25d-b93afb1701ff" (UID: "a49682b8-c9c8-4d9f-a25d-b93afb1701ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:19:01.552542 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.552292    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a49682b8-c9c8-4d9f-a25d-b93afb1701ff" (UID: "a49682b8-c9c8-4d9f-a25d-b93afb1701ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:19:01.553880 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.553851    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kube-api-access-dh4sv" (OuterVolumeSpecName: "kube-api-access-dh4sv") pod "a49682b8-c9c8-4d9f-a25d-b93afb1701ff" (UID: "a49682b8-c9c8-4d9f-a25d-b93afb1701ff"). InnerVolumeSpecName "kube-api-access-dh4sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:19:01.554012 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.553988    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a49682b8-c9c8-4d9f-a25d-b93afb1701ff" (UID: "a49682b8-c9c8-4d9f-a25d-b93afb1701ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:19:01.652598 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.652564    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:19:01.652598 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.652590    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-cabundle-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:19:01.652598 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.652600    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kserve-provision-location\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:19:01.652598 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.652609    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-proxy-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:19:01.652914 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.652618    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh4sv\" (UniqueName: \"kubernetes.io/projected/a49682b8-c9c8-4d9f-a25d-b93afb1701ff-kube-api-access-dh4sv\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:19:01.657217 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.657185    2581 generic.go:356] "Generic (PLEG): container finished" podID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerID="1c9cf60fec0cad1877db590e6d926435c7d6f8b90addcd951dcad4eebd772439" exitCode=0
Jul 07 15:19:01.657373 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.657250    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"
Jul 07 15:19:01.657373 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.657268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" event={"ID":"a49682b8-c9c8-4d9f-a25d-b93afb1701ff","Type":"ContainerDied","Data":"1c9cf60fec0cad1877db590e6d926435c7d6f8b90addcd951dcad4eebd772439"}
Jul 07 15:19:01.657473 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.657385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5" event={"ID":"a49682b8-c9c8-4d9f-a25d-b93afb1701ff","Type":"ContainerDied","Data":"b22ab17e6427bab27fbb78cef54badd779be118f47dec60ffdac993d1d7487cd"}
Jul 07 15:19:01.657473 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.657415    2581 scope.go:117] "RemoveContainer" containerID="b00e8b7432246bcb76373e9461d726071eabab47c1a1d8d93d8064d4543d20de"
Jul 07 15:19:01.666992 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.666748    2581 scope.go:117] "RemoveContainer" containerID="1c9cf60fec0cad1877db590e6d926435c7d6f8b90addcd951dcad4eebd772439"
Jul 07 15:19:01.675832 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.675812    2581 scope.go:117] "RemoveContainer" containerID="ea2a588e19f36380c6bb0fa1939d7f5f0effca0cb1b0375f097d0d03acbaf54d"
Jul 07 15:19:01.683865 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.683843    2581 scope.go:117] "RemoveContainer" containerID="b00e8b7432246bcb76373e9461d726071eabab47c1a1d8d93d8064d4543d20de"
Jul 07 15:19:01.684200 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:19:01.684179    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00e8b7432246bcb76373e9461d726071eabab47c1a1d8d93d8064d4543d20de\": container with ID starting with b00e8b7432246bcb76373e9461d726071eabab47c1a1d8d93d8064d4543d20de not found: ID does not exist" containerID="b00e8b7432246bcb76373e9461d726071eabab47c1a1d8d93d8064d4543d20de"
Jul 07 15:19:01.684255 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.684211    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00e8b7432246bcb76373e9461d726071eabab47c1a1d8d93d8064d4543d20de"} err="failed to get container status \"b00e8b7432246bcb76373e9461d726071eabab47c1a1d8d93d8064d4543d20de\": rpc error: code = NotFound desc = could not find container \"b00e8b7432246bcb76373e9461d726071eabab47c1a1d8d93d8064d4543d20de\": container with ID starting with b00e8b7432246bcb76373e9461d726071eabab47c1a1d8d93d8064d4543d20de not found: ID does not exist"
Jul 07 15:19:01.684255 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.684234    2581 scope.go:117] "RemoveContainer" containerID="1c9cf60fec0cad1877db590e6d926435c7d6f8b90addcd951dcad4eebd772439"
Jul 07 15:19:01.684580 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:19:01.684552    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9cf60fec0cad1877db590e6d926435c7d6f8b90addcd951dcad4eebd772439\": container with ID starting with 1c9cf60fec0cad1877db590e6d926435c7d6f8b90addcd951dcad4eebd772439 not found: ID does not exist" containerID="1c9cf60fec0cad1877db590e6d926435c7d6f8b90addcd951dcad4eebd772439"
Jul 07 15:19:01.684660 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.684588    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9cf60fec0cad1877db590e6d926435c7d6f8b90addcd951dcad4eebd772439"} err="failed to get container status \"1c9cf60fec0cad1877db590e6d926435c7d6f8b90addcd951dcad4eebd772439\": rpc error: code = NotFound desc = could not find container \"1c9cf60fec0cad1877db590e6d926435c7d6f8b90addcd951dcad4eebd772439\": container with ID starting with 1c9cf60fec0cad1877db590e6d926435c7d6f8b90addcd951dcad4eebd772439 not found: ID does not exist"
Jul 07 15:19:01.684660 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.684604    2581 scope.go:117] "RemoveContainer" containerID="ea2a588e19f36380c6bb0fa1939d7f5f0effca0cb1b0375f097d0d03acbaf54d"
Jul 07 15:19:01.684855 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:19:01.684836    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2a588e19f36380c6bb0fa1939d7f5f0effca0cb1b0375f097d0d03acbaf54d\": container with ID starting with ea2a588e19f36380c6bb0fa1939d7f5f0effca0cb1b0375f097d0d03acbaf54d not found: ID does not exist" containerID="ea2a588e19f36380c6bb0fa1939d7f5f0effca0cb1b0375f097d0d03acbaf54d"
Jul 07 15:19:01.684928 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.684860    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2a588e19f36380c6bb0fa1939d7f5f0effca0cb1b0375f097d0d03acbaf54d"} err="failed to get container status \"ea2a588e19f36380c6bb0fa1939d7f5f0effca0cb1b0375f097d0d03acbaf54d\": rpc error: code = NotFound desc = could not find container \"ea2a588e19f36380c6bb0fa1939d7f5f0effca0cb1b0375f097d0d03acbaf54d\": container with ID starting with ea2a588e19f36380c6bb0fa1939d7f5f0effca0cb1b0375f097d0d03acbaf54d not found: ID does not exist"
Jul 07 15:19:01.691072 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.691040    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"]
Jul 07 15:19:01.697753 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.697710    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d3e813-predictor-5ccd495944-jplx5"]
Jul 07 15:19:01.751210 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.751127    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"]
Jul 07 15:19:01.751488 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.751475    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="storage-initializer"
Jul 07 15:19:01.751540 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.751489    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="storage-initializer"
Jul 07 15:19:01.751540 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.751498    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kube-rbac-proxy"
Jul 07 15:19:01.751540 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.751504    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kube-rbac-proxy"
Jul 07 15:19:01.751540 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.751519    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a27b72f-4351-45a7-8978-2770feeb7d7f" containerName="storage-initializer"
Jul 07 15:19:01.751540 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.751525    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a27b72f-4351-45a7-8978-2770feeb7d7f" containerName="storage-initializer"
Jul 07 15:19:01.751540 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.751532    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kserve-container"
Jul 07 15:19:01.751540 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.751537    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kserve-container"
Jul 07 15:19:01.751748 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.751543    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a27b72f-4351-45a7-8978-2770feeb7d7f" containerName="storage-initializer"
Jul 07 15:19:01.751748 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.751549    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a27b72f-4351-45a7-8978-2770feeb7d7f" containerName="storage-initializer"
Jul 07 15:19:01.751748 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.751604    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a27b72f-4351-45a7-8978-2770feeb7d7f" containerName="storage-initializer"
Jul 07 15:19:01.751748 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.751613    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kserve-container"
Jul 07 15:19:01.751748 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.751620    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" containerName="kube-rbac-proxy"
Jul 07 15:19:01.751748 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.751719    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a27b72f-4351-45a7-8978-2770feeb7d7f" containerName="storage-initializer"
Jul 07 15:19:01.756734 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.756709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:01.759770 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.759747    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-89de80-predictor-serving-cert\""
Jul 07 15:19:01.760278 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.760258    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-89de80-kube-rbac-proxy-sar-config\""
Jul 07 15:19:01.760451 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.760265    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 15:19:01.760664 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.760650    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 15:19:01.760866 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.760851    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-89de80-dockercfg-tzd4b\""
Jul 07 15:19:01.760939 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.760918    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-89de80\""
Jul 07 15:19:01.760983 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.760939    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 15:19:01.767434 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.767404    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"]
Jul 07 15:19:01.854461 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.854418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-cabundle-cert\") pod \"isvc-init-fail-89de80-predictor-555565cf89-qcg8j\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") " pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:01.854658 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.854477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l8dw\" (UniqueName: \"kubernetes.io/projected/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kube-api-access-8l8dw\") pod \"isvc-init-fail-89de80-predictor-555565cf89-qcg8j\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") " pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:01.854658 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.854521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kserve-provision-location\") pod \"isvc-init-fail-89de80-predictor-555565cf89-qcg8j\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") " pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:01.854658 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.854546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-89de80-predictor-555565cf89-qcg8j\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") " pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:01.854807 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.854659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-proxy-tls\") pod \"isvc-init-fail-89de80-predictor-555565cf89-qcg8j\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") " pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:01.956105 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.956065    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-proxy-tls\") pod \"isvc-init-fail-89de80-predictor-555565cf89-qcg8j\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") " pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:01.956298 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.956127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-cabundle-cert\") pod \"isvc-init-fail-89de80-predictor-555565cf89-qcg8j\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") " pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:01.956298 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.956171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8dw\" (UniqueName: \"kubernetes.io/projected/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kube-api-access-8l8dw\") pod \"isvc-init-fail-89de80-predictor-555565cf89-qcg8j\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") " pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:01.956298 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.956210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kserve-provision-location\") pod \"isvc-init-fail-89de80-predictor-555565cf89-qcg8j\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") " pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:01.956512 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.956338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-89de80-predictor-555565cf89-qcg8j\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") " pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:01.956666 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.956648    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kserve-provision-location\") pod \"isvc-init-fail-89de80-predictor-555565cf89-qcg8j\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") " pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:01.957013 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.956994    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-cabundle-cert\") pod \"isvc-init-fail-89de80-predictor-555565cf89-qcg8j\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") " pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:01.957062 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.956997    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-89de80-predictor-555565cf89-qcg8j\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") " pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:01.958613 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.958596    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-proxy-tls\") pod \"isvc-init-fail-89de80-predictor-555565cf89-qcg8j\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") " pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:01.964786 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:01.964767    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l8dw\" (UniqueName: \"kubernetes.io/projected/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kube-api-access-8l8dw\") pod \"isvc-init-fail-89de80-predictor-555565cf89-qcg8j\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") " pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:02.068877 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:02.068782    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:02.214528 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:02.214375    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"]
Jul 07 15:19:02.217248 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:19:02.217219    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea47f1f4_0c6e_475f_b8c9_84404de6dbd5.slice/crio-3882589a5a0bd872e13af218ab5bc681821a01c9267c7ad6dd974b7b9ef392c4 WatchSource:0}: Error finding container 3882589a5a0bd872e13af218ab5bc681821a01c9267c7ad6dd974b7b9ef392c4: Status 404 returned error can't find the container with id 3882589a5a0bd872e13af218ab5bc681821a01c9267c7ad6dd974b7b9ef392c4
Jul 07 15:19:02.665134 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:02.665089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j" event={"ID":"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5","Type":"ContainerStarted","Data":"8523fb2054e9d478e15cbcde90c133e889fef5c58896cfda0342fb8e6814a0c6"}
Jul 07 15:19:02.665342 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:02.665164    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j" event={"ID":"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5","Type":"ContainerStarted","Data":"3882589a5a0bd872e13af218ab5bc681821a01c9267c7ad6dd974b7b9ef392c4"}
Jul 07 15:19:02.933172 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:02.933087    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a49682b8-c9c8-4d9f-a25d-b93afb1701ff" path="/var/lib/kubelet/pods/a49682b8-c9c8-4d9f-a25d-b93afb1701ff/volumes"
Jul 07 15:19:06.684648 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:06.684619    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-89de80-predictor-555565cf89-qcg8j_ea47f1f4-0c6e-475f-b8c9-84404de6dbd5/storage-initializer/0.log"
Jul 07 15:19:06.685032 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:06.684658    2581 generic.go:356] "Generic (PLEG): container finished" podID="ea47f1f4-0c6e-475f-b8c9-84404de6dbd5" containerID="8523fb2054e9d478e15cbcde90c133e889fef5c58896cfda0342fb8e6814a0c6" exitCode=1
Jul 07 15:19:06.685032 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:06.684734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j" event={"ID":"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5","Type":"ContainerDied","Data":"8523fb2054e9d478e15cbcde90c133e889fef5c58896cfda0342fb8e6814a0c6"}
Jul 07 15:19:07.691022 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:07.690996    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-89de80-predictor-555565cf89-qcg8j_ea47f1f4-0c6e-475f-b8c9-84404de6dbd5/storage-initializer/0.log"
Jul 07 15:19:07.691451 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:07.691117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j" event={"ID":"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5","Type":"ContainerStarted","Data":"1bb249adefe54fc7ece909ab839d054e4d4263c77351a02e7437942fb192dcb5"}
Jul 07 15:19:09.702337 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:09.702313    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-89de80-predictor-555565cf89-qcg8j_ea47f1f4-0c6e-475f-b8c9-84404de6dbd5/storage-initializer/1.log"
Jul 07 15:19:09.702800 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:09.702714    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-89de80-predictor-555565cf89-qcg8j_ea47f1f4-0c6e-475f-b8c9-84404de6dbd5/storage-initializer/0.log"
Jul 07 15:19:09.702800 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:09.702746    2581 generic.go:356] "Generic (PLEG): container finished" podID="ea47f1f4-0c6e-475f-b8c9-84404de6dbd5" containerID="1bb249adefe54fc7ece909ab839d054e4d4263c77351a02e7437942fb192dcb5" exitCode=1
Jul 07 15:19:09.702882 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:09.702816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j" event={"ID":"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5","Type":"ContainerDied","Data":"1bb249adefe54fc7ece909ab839d054e4d4263c77351a02e7437942fb192dcb5"}
Jul 07 15:19:09.702882 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:09.702877    2581 scope.go:117] "RemoveContainer" containerID="8523fb2054e9d478e15cbcde90c133e889fef5c58896cfda0342fb8e6814a0c6"
Jul 07 15:19:09.703213 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:09.703187    2581 scope.go:117] "RemoveContainer" containerID="8523fb2054e9d478e15cbcde90c133e889fef5c58896cfda0342fb8e6814a0c6"
Jul 07 15:19:09.713686 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:19:09.713662    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-89de80-predictor-555565cf89-qcg8j_kserve-ci-e2e-test(ea47f1f4-0c6e-475f-b8c9-84404de6dbd5)\"" pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j" podUID="ea47f1f4-0c6e-475f-b8c9-84404de6dbd5"
Jul 07 15:19:09.713811 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:19:09.713784    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-89de80-predictor-555565cf89-qcg8j_kserve-ci-e2e-test_ea47f1f4-0c6e-475f-b8c9-84404de6dbd5_0 in pod sandbox 3882589a5a0bd872e13af218ab5bc681821a01c9267c7ad6dd974b7b9ef392c4 from index: no such id: '8523fb2054e9d478e15cbcde90c133e889fef5c58896cfda0342fb8e6814a0c6'" containerID="8523fb2054e9d478e15cbcde90c133e889fef5c58896cfda0342fb8e6814a0c6"
Jul 07 15:19:09.713859 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:09.713822    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8523fb2054e9d478e15cbcde90c133e889fef5c58896cfda0342fb8e6814a0c6"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-89de80-predictor-555565cf89-qcg8j_kserve-ci-e2e-test_ea47f1f4-0c6e-475f-b8c9-84404de6dbd5_0 in pod sandbox 3882589a5a0bd872e13af218ab5bc681821a01c9267c7ad6dd974b7b9ef392c4 from index: no such id: '8523fb2054e9d478e15cbcde90c133e889fef5c58896cfda0342fb8e6814a0c6'"
Jul 07 15:19:10.708490 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:10.708461    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-89de80-predictor-555565cf89-qcg8j_ea47f1f4-0c6e-475f-b8c9-84404de6dbd5/storage-initializer/1.log"
Jul 07 15:19:11.749548 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:11.749513    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"]
Jul 07 15:19:11.883048 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:11.883025    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-89de80-predictor-555565cf89-qcg8j_ea47f1f4-0c6e-475f-b8c9-84404de6dbd5/storage-initializer/1.log"
Jul 07 15:19:11.886433 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:11.886417    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:11.938765 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:11.938731    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-cabundle-cert\") pod \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") "
Jul 07 15:19:11.938980 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:11.938772    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l8dw\" (UniqueName: \"kubernetes.io/projected/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kube-api-access-8l8dw\") pod \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") "
Jul 07 15:19:11.938980 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:11.938816    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kserve-provision-location\") pod \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") "
Jul 07 15:19:11.938980 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:11.938953    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kube-rbac-proxy-sar-config\") pod \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") "
Jul 07 15:19:11.939154 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:11.939025    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-proxy-tls\") pod \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\" (UID: \"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5\") "
Jul 07 15:19:11.939154 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:11.939068    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea47f1f4-0c6e-475f-b8c9-84404de6dbd5" (UID: "ea47f1f4-0c6e-475f-b8c9-84404de6dbd5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:19:11.939255 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:11.939213    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea47f1f4-0c6e-475f-b8c9-84404de6dbd5" (UID: "ea47f1f4-0c6e-475f-b8c9-84404de6dbd5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:19:11.939312 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:11.939289    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ea47f1f4-0c6e-475f-b8c9-84404de6dbd5" (UID: "ea47f1f4-0c6e-475f-b8c9-84404de6dbd5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:19:11.939386 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:11.939301    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-cabundle-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:19:11.939386 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:11.939333    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kserve-provision-location\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:19:11.940937 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:11.940910    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kube-api-access-8l8dw" (OuterVolumeSpecName: "kube-api-access-8l8dw") pod "ea47f1f4-0c6e-475f-b8c9-84404de6dbd5" (UID: "ea47f1f4-0c6e-475f-b8c9-84404de6dbd5"). InnerVolumeSpecName "kube-api-access-8l8dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:19:11.941028 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:11.940961    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea47f1f4-0c6e-475f-b8c9-84404de6dbd5" (UID: "ea47f1f4-0c6e-475f-b8c9-84404de6dbd5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:19:12.040954 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:12.040848    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l8dw\" (UniqueName: \"kubernetes.io/projected/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kube-api-access-8l8dw\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:19:12.040954 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:12.040891    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:19:12.040954 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:12.040907    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5-proxy-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:19:12.717941 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:12.717910    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-89de80-predictor-555565cf89-qcg8j_ea47f1f4-0c6e-475f-b8c9-84404de6dbd5/storage-initializer/1.log"
Jul 07 15:19:12.718116 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:12.718006    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"
Jul 07 15:19:12.718116 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:12.718009    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j" event={"ID":"ea47f1f4-0c6e-475f-b8c9-84404de6dbd5","Type":"ContainerDied","Data":"3882589a5a0bd872e13af218ab5bc681821a01c9267c7ad6dd974b7b9ef392c4"}
Jul 07 15:19:12.718116 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:12.718083    2581 scope.go:117] "RemoveContainer" containerID="1bb249adefe54fc7ece909ab839d054e4d4263c77351a02e7437942fb192dcb5"
Jul 07 15:19:12.755073 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:12.755041    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"]
Jul 07 15:19:12.759622 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:12.759591    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-89de80-predictor-555565cf89-qcg8j"]
Jul 07 15:19:12.931962 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:12.931927    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea47f1f4-0c6e-475f-b8c9-84404de6dbd5" path="/var/lib/kubelet/pods/ea47f1f4-0c6e-475f-b8c9-84404de6dbd5/volumes"
Jul 07 15:19:16.940901 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.940873    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"]
Jul 07 15:19:16.941384 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.941218    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea47f1f4-0c6e-475f-b8c9-84404de6dbd5" containerName="storage-initializer"
Jul 07 15:19:16.941384 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.941231    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea47f1f4-0c6e-475f-b8c9-84404de6dbd5" containerName="storage-initializer"
Jul 07 15:19:16.941384 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.941238    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea47f1f4-0c6e-475f-b8c9-84404de6dbd5" containerName="storage-initializer"
Jul 07 15:19:16.941384 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.941243    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea47f1f4-0c6e-475f-b8c9-84404de6dbd5" containerName="storage-initializer"
Jul 07 15:19:16.941384 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.941303    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea47f1f4-0c6e-475f-b8c9-84404de6dbd5" containerName="storage-initializer"
Jul 07 15:19:16.941577 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.941439    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea47f1f4-0c6e-475f-b8c9-84404de6dbd5" containerName="storage-initializer"
Jul 07 15:19:16.945672 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.945655    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:16.948622 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.948596    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-2e704-predictor-serving-cert\""
Jul 07 15:19:16.948622 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.948596    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-2e704-kube-rbac-proxy-sar-config\""
Jul 07 15:19:16.948795 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.948713    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 15:19:16.950267 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.950248    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 15:19:16.950370 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.950274    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 15:19:16.950370 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.950253    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 15:19:16.950370 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.950247    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z5ftp\""
Jul 07 15:19:16.956790 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:16.956764    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"]
Jul 07 15:19:17.083429 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.083395    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsr4h\" (UniqueName: \"kubernetes.io/projected/f3baa47e-215e-4f27-81d4-7874722cfdec-kube-api-access-vsr4h\") pod \"raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") " pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.083627 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.083456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3baa47e-215e-4f27-81d4-7874722cfdec-cabundle-cert\") pod \"raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") " pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.083627 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.083486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3baa47e-215e-4f27-81d4-7874722cfdec-proxy-tls\") pod \"raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") " pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.083627 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.083526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3baa47e-215e-4f27-81d4-7874722cfdec-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") " pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.083627 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.083558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3baa47e-215e-4f27-81d4-7874722cfdec-kserve-provision-location\") pod \"raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") " pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.190832 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.190795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3baa47e-215e-4f27-81d4-7874722cfdec-cabundle-cert\") pod \"raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") " pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.191045 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.190848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3baa47e-215e-4f27-81d4-7874722cfdec-proxy-tls\") pod \"raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") " pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.191045 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.190897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3baa47e-215e-4f27-81d4-7874722cfdec-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") " pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.191045 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.190935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3baa47e-215e-4f27-81d4-7874722cfdec-kserve-provision-location\") pod \"raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") " pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.191512 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.191385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsr4h\" (UniqueName: \"kubernetes.io/projected/f3baa47e-215e-4f27-81d4-7874722cfdec-kube-api-access-vsr4h\") pod \"raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") " pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.192167 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.192141    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3baa47e-215e-4f27-81d4-7874722cfdec-kserve-provision-location\") pod \"raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") " pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.192538 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.192514    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3baa47e-215e-4f27-81d4-7874722cfdec-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") " pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.192621 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.192570    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3baa47e-215e-4f27-81d4-7874722cfdec-cabundle-cert\") pod \"raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") " pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.194184 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.194164    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3baa47e-215e-4f27-81d4-7874722cfdec-proxy-tls\") pod \"raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") " pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.202452 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.202430    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsr4h\" (UniqueName: \"kubernetes.io/projected/f3baa47e-215e-4f27-81d4-7874722cfdec-kube-api-access-vsr4h\") pod \"raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") " pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.257028 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.256988    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:17.399491 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.399465    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"]
Jul 07 15:19:17.402016 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:19:17.401989    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3baa47e_215e_4f27_81d4_7874722cfdec.slice/crio-bf0522ce87816338e197265e2384c0733337a13dbaa00d1c24b60c019c47d311 WatchSource:0}: Error finding container bf0522ce87816338e197265e2384c0733337a13dbaa00d1c24b60c019c47d311: Status 404 returned error can't find the container with id bf0522ce87816338e197265e2384c0733337a13dbaa00d1c24b60c019c47d311
Jul 07 15:19:17.741625 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.741516    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" event={"ID":"f3baa47e-215e-4f27-81d4-7874722cfdec","Type":"ContainerStarted","Data":"300af35aafbdfc70c91b45a51a5680ffe956c4485b07c8c1c443807027791a98"}
Jul 07 15:19:17.741625 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:17.741582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" event={"ID":"f3baa47e-215e-4f27-81d4-7874722cfdec","Type":"ContainerStarted","Data":"bf0522ce87816338e197265e2384c0733337a13dbaa00d1c24b60c019c47d311"}
Jul 07 15:19:21.759093 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:21.759057    2581 generic.go:356] "Generic (PLEG): container finished" podID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerID="300af35aafbdfc70c91b45a51a5680ffe956c4485b07c8c1c443807027791a98" exitCode=0
Jul 07 15:19:21.759495 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:21.759130    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" event={"ID":"f3baa47e-215e-4f27-81d4-7874722cfdec","Type":"ContainerDied","Data":"300af35aafbdfc70c91b45a51a5680ffe956c4485b07c8c1c443807027791a98"}
Jul 07 15:19:22.766441 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:22.766408    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" event={"ID":"f3baa47e-215e-4f27-81d4-7874722cfdec","Type":"ContainerStarted","Data":"95486e900fafe1f1ed144dde7bcabf2ae13ca9c4152473ad9f74e06daab59c87"}
Jul 07 15:19:22.766832 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:22.766474    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" event={"ID":"f3baa47e-215e-4f27-81d4-7874722cfdec","Type":"ContainerStarted","Data":"bb234b2d348a613bf7b7753909893ef0614e501b313da767a9996d2bfb59b68c"}
Jul 07 15:19:22.766832 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:22.766650    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:22.789994 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:22.789943    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" podStartSLOduration=3.084480791 podStartE2EDuration="6.789926721s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.70544593s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:19:22.788387754 +0000 UTC m=+1324.521000074" watchObservedRunningTime="2026-07-07 15:19:22.789926721 +0000 UTC m=+1324.522539040"
Jul 07 15:19:23.770970 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:23.770927    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:23.772434 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:23.772406    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 15:19:24.775506 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:24.775462    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 15:19:29.779958 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:29.779929    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:19:29.780548 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:29.780520    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 15:19:39.781403 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:39.781339    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 15:19:49.780675 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:49.780591    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 15:19:59.780633 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:19:59.780595    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 15:20:09.781368 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:09.781307    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 15:20:19.781412 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:19.781369    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 15:20:29.781247 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:29.781206    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 15:20:39.781342 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:39.781312    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:20:47.320444 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:47.320409    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"]
Jul 07 15:20:47.320994 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:47.320631    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kserve-container" containerID="cri-o://bb234b2d348a613bf7b7753909893ef0614e501b313da767a9996d2bfb59b68c" gracePeriod=30
Jul 07 15:20:47.320994 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:47.320679    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kube-rbac-proxy" containerID="cri-o://95486e900fafe1f1ed144dde7bcabf2ae13ca9c4152473ad9f74e06daab59c87" gracePeriod=30
Jul 07 15:20:48.113700 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:48.113665    2581 generic.go:356] "Generic (PLEG): container finished" podID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerID="95486e900fafe1f1ed144dde7bcabf2ae13ca9c4152473ad9f74e06daab59c87" exitCode=2
Jul 07 15:20:48.113882 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:48.113739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" event={"ID":"f3baa47e-215e-4f27-81d4-7874722cfdec","Type":"ContainerDied","Data":"95486e900fafe1f1ed144dde7bcabf2ae13ca9c4152473ad9f74e06daab59c87"}
Jul 07 15:20:49.776758 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:49.776706    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 07 15:20:49.781069 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:49.781040    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 15:20:52.134228 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.134195    2581 generic.go:356] "Generic (PLEG): container finished" podID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerID="bb234b2d348a613bf7b7753909893ef0614e501b313da767a9996d2bfb59b68c" exitCode=0
Jul 07 15:20:52.134593 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.134270    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" event={"ID":"f3baa47e-215e-4f27-81d4-7874722cfdec","Type":"ContainerDied","Data":"bb234b2d348a613bf7b7753909893ef0614e501b313da767a9996d2bfb59b68c"}
Jul 07 15:20:52.168747 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.168716    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:20:52.250122 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.250088    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsr4h\" (UniqueName: \"kubernetes.io/projected/f3baa47e-215e-4f27-81d4-7874722cfdec-kube-api-access-vsr4h\") pod \"f3baa47e-215e-4f27-81d4-7874722cfdec\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") "
Jul 07 15:20:52.250304 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.250151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3baa47e-215e-4f27-81d4-7874722cfdec-kserve-provision-location\") pod \"f3baa47e-215e-4f27-81d4-7874722cfdec\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") "
Jul 07 15:20:52.250304 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.250263    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3baa47e-215e-4f27-81d4-7874722cfdec-proxy-tls\") pod \"f3baa47e-215e-4f27-81d4-7874722cfdec\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") "
Jul 07 15:20:52.250304 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.250295    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3baa47e-215e-4f27-81d4-7874722cfdec-kube-rbac-proxy-sar-config\") pod \"f3baa47e-215e-4f27-81d4-7874722cfdec\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") "
Jul 07 15:20:52.250486 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.250332    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3baa47e-215e-4f27-81d4-7874722cfdec-cabundle-cert\") pod \"f3baa47e-215e-4f27-81d4-7874722cfdec\" (UID: \"f3baa47e-215e-4f27-81d4-7874722cfdec\") "
Jul 07 15:20:52.250546 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.250500    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3baa47e-215e-4f27-81d4-7874722cfdec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3baa47e-215e-4f27-81d4-7874722cfdec" (UID: "f3baa47e-215e-4f27-81d4-7874722cfdec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:20:52.250652 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.250631    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3baa47e-215e-4f27-81d4-7874722cfdec-kserve-provision-location\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:20:52.250740 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.250724    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3baa47e-215e-4f27-81d4-7874722cfdec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f3baa47e-215e-4f27-81d4-7874722cfdec" (UID: "f3baa47e-215e-4f27-81d4-7874722cfdec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:20:52.250796 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.250753    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3baa47e-215e-4f27-81d4-7874722cfdec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f3baa47e-215e-4f27-81d4-7874722cfdec" (UID: "f3baa47e-215e-4f27-81d4-7874722cfdec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:20:52.252341 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.252318    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3baa47e-215e-4f27-81d4-7874722cfdec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f3baa47e-215e-4f27-81d4-7874722cfdec" (UID: "f3baa47e-215e-4f27-81d4-7874722cfdec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:20:52.252341 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.252328    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3baa47e-215e-4f27-81d4-7874722cfdec-kube-api-access-vsr4h" (OuterVolumeSpecName: "kube-api-access-vsr4h") pod "f3baa47e-215e-4f27-81d4-7874722cfdec" (UID: "f3baa47e-215e-4f27-81d4-7874722cfdec"). InnerVolumeSpecName "kube-api-access-vsr4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:20:52.338783 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.338702    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"]
Jul 07 15:20:52.339096 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.339083    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kube-rbac-proxy"
Jul 07 15:20:52.339140 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.339098    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kube-rbac-proxy"
Jul 07 15:20:52.339140 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.339113    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="storage-initializer"
Jul 07 15:20:52.339140 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.339119    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="storage-initializer"
Jul 07 15:20:52.339140 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.339133    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kserve-container"
Jul 07 15:20:52.339265 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.339142    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kserve-container"
Jul 07 15:20:52.339265 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.339191    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kube-rbac-proxy"
Jul 07 15:20:52.339265 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.339201    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" containerName="kserve-container"
Jul 07 15:20:52.342547 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.342531    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.348576 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.348552    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-00f07-kube-rbac-proxy-sar-config\""
Jul 07 15:20:52.348708 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.348660    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-00f07-predictor-serving-cert\""
Jul 07 15:20:52.351919 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.351899    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3baa47e-215e-4f27-81d4-7874722cfdec-proxy-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:20:52.351919 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.351919    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3baa47e-215e-4f27-81d4-7874722cfdec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:20:52.352068 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.351929    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3baa47e-215e-4f27-81d4-7874722cfdec-cabundle-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:20:52.352068 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.351938    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsr4h\" (UniqueName: \"kubernetes.io/projected/f3baa47e-215e-4f27-81d4-7874722cfdec-kube-api-access-vsr4h\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:20:52.355382 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.355343    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"]
Jul 07 15:20:52.453255 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.453210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-proxy-tls\") pod \"raw-sklearn-runtime-00f07-predictor-5977594885-pc922\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.453444 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.453291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttrk8\" (UniqueName: \"kubernetes.io/projected/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kube-api-access-ttrk8\") pod \"raw-sklearn-runtime-00f07-predictor-5977594885-pc922\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.453444 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.453328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-00f07-predictor-5977594885-pc922\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.453444 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.453400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-cabundle-cert\") pod \"raw-sklearn-runtime-00f07-predictor-5977594885-pc922\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.453444 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.453441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kserve-provision-location\") pod \"raw-sklearn-runtime-00f07-predictor-5977594885-pc922\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.554527 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.554496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-proxy-tls\") pod \"raw-sklearn-runtime-00f07-predictor-5977594885-pc922\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.554716 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.554546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttrk8\" (UniqueName: \"kubernetes.io/projected/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kube-api-access-ttrk8\") pod \"raw-sklearn-runtime-00f07-predictor-5977594885-pc922\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.554716 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.554574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-00f07-predictor-5977594885-pc922\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.554716 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.554607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-cabundle-cert\") pod \"raw-sklearn-runtime-00f07-predictor-5977594885-pc922\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.554716 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.554653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kserve-provision-location\") pod \"raw-sklearn-runtime-00f07-predictor-5977594885-pc922\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.555100 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.555079    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kserve-provision-location\") pod \"raw-sklearn-runtime-00f07-predictor-5977594885-pc922\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.555450 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.555429    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-cabundle-cert\") pod \"raw-sklearn-runtime-00f07-predictor-5977594885-pc922\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.555502 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.555429    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-00f07-predictor-5977594885-pc922\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.556893 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.556876    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-proxy-tls\") pod \"raw-sklearn-runtime-00f07-predictor-5977594885-pc922\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.564143 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.564121    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttrk8\" (UniqueName: \"kubernetes.io/projected/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kube-api-access-ttrk8\") pod \"raw-sklearn-runtime-00f07-predictor-5977594885-pc922\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.653416 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.653290    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:52.790251 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:52.790222    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"]
Jul 07 15:20:52.792707 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:20:52.792682    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a1ebc8_7d5e_4f82_9591_e20585bff1cf.slice/crio-ece10dbd668176e9886f497fc8b738e7e3c62c904b462fa4292d5a054aa82f8d WatchSource:0}: Error finding container ece10dbd668176e9886f497fc8b738e7e3c62c904b462fa4292d5a054aa82f8d: Status 404 returned error can't find the container with id ece10dbd668176e9886f497fc8b738e7e3c62c904b462fa4292d5a054aa82f8d
Jul 07 15:20:53.140088 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:53.140053    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"
Jul 07 15:20:53.140580 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:53.140052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m" event={"ID":"f3baa47e-215e-4f27-81d4-7874722cfdec","Type":"ContainerDied","Data":"bf0522ce87816338e197265e2384c0733337a13dbaa00d1c24b60c019c47d311"}
Jul 07 15:20:53.140580 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:53.140235    2581 scope.go:117] "RemoveContainer" containerID="95486e900fafe1f1ed144dde7bcabf2ae13ca9c4152473ad9f74e06daab59c87"
Jul 07 15:20:53.141616 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:53.141593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" event={"ID":"69a1ebc8-7d5e-4f82-9591-e20585bff1cf","Type":"ContainerStarted","Data":"1a1ecdb35f3a88ce82cc1219347c79a6643d83c733b0128bbe7c9decb4b69298"}
Jul 07 15:20:53.141682 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:53.141653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" event={"ID":"69a1ebc8-7d5e-4f82-9591-e20585bff1cf","Type":"ContainerStarted","Data":"ece10dbd668176e9886f497fc8b738e7e3c62c904b462fa4292d5a054aa82f8d"}
Jul 07 15:20:53.148779 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:53.148758    2581 scope.go:117] "RemoveContainer" containerID="bb234b2d348a613bf7b7753909893ef0614e501b313da767a9996d2bfb59b68c"
Jul 07 15:20:53.156845 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:53.156822    2581 scope.go:117] "RemoveContainer" containerID="300af35aafbdfc70c91b45a51a5680ffe956c4485b07c8c1c443807027791a98"
Jul 07 15:20:53.194454 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:53.194370    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"]
Jul 07 15:20:53.195803 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:53.195779    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2e704-predictor-bb8b4dfb5-7b52m"]
Jul 07 15:20:54.935139 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:54.935102    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3baa47e-215e-4f27-81d4-7874722cfdec" path="/var/lib/kubelet/pods/f3baa47e-215e-4f27-81d4-7874722cfdec/volumes"
Jul 07 15:20:57.165139 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:57.165100    2581 generic.go:356] "Generic (PLEG): container finished" podID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerID="1a1ecdb35f3a88ce82cc1219347c79a6643d83c733b0128bbe7c9decb4b69298" exitCode=0
Jul 07 15:20:57.165572 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:57.165181    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" event={"ID":"69a1ebc8-7d5e-4f82-9591-e20585bff1cf","Type":"ContainerDied","Data":"1a1ecdb35f3a88ce82cc1219347c79a6643d83c733b0128bbe7c9decb4b69298"}
Jul 07 15:20:58.171134 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:58.171099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" event={"ID":"69a1ebc8-7d5e-4f82-9591-e20585bff1cf","Type":"ContainerStarted","Data":"14c897d5662a1f52f533ec5df307bc981e6139dca7759f912af2b84780eb65d8"}
Jul 07 15:20:58.171613 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:58.171162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" event={"ID":"69a1ebc8-7d5e-4f82-9591-e20585bff1cf","Type":"ContainerStarted","Data":"8531040f2fbad7184c0f7b2ffa6acd3430cf7f4e7e34456fade970bef92ebd79"}
Jul 07 15:20:58.171613 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:58.171382    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:58.200413 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:58.200339    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" podStartSLOduration=2.666464817 podStartE2EDuration="6.200322695s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.533857878s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:20:58.198323905 +0000 UTC m=+1419.930936225" watchObservedRunningTime="2026-07-07 15:20:58.200322695 +0000 UTC m=+1419.932935014"
Jul 07 15:20:59.181366 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:59.181314    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:20:59.182932 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:20:59.182900    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 15:21:00.185910 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:21:00.185868    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 15:21:05.191215 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:21:05.191182    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:21:05.191744 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:21:05.191718    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 15:21:15.191987 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:21:15.191951    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 15:21:25.192311 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:21:25.192271    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 15:21:35.192242 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:21:35.192198    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 15:21:45.191742 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:21:45.191696    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 15:21:55.192323 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:21:55.192283    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 15:22:05.192177 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:05.192134    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 15:22:15.193223 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:15.193189    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:22:22.441552 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:22.441517    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"]
Jul 07 15:22:22.441982 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:22.441810    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kserve-container" containerID="cri-o://8531040f2fbad7184c0f7b2ffa6acd3430cf7f4e7e34456fade970bef92ebd79" gracePeriod=30
Jul 07 15:22:22.442105 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:22.442030    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kube-rbac-proxy" containerID="cri-o://14c897d5662a1f52f533ec5df307bc981e6139dca7759f912af2b84780eb65d8" gracePeriod=30
Jul 07 15:22:23.537392 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:23.537343    2581 generic.go:356] "Generic (PLEG): container finished" podID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerID="14c897d5662a1f52f533ec5df307bc981e6139dca7759f912af2b84780eb65d8" exitCode=2
Jul 07 15:22:23.537771 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:23.537413    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" event={"ID":"69a1ebc8-7d5e-4f82-9591-e20585bff1cf","Type":"ContainerDied","Data":"14c897d5662a1f52f533ec5df307bc981e6139dca7759f912af2b84780eb65d8"}
Jul 07 15:22:25.186713 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:25.186673    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 07 15:22:25.192007 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:25.191983    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 15:22:27.186702 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.186677    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:22:27.278748 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.278662    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttrk8\" (UniqueName: \"kubernetes.io/projected/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kube-api-access-ttrk8\") pod \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") "
Jul 07 15:22:27.278748 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.278711    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-proxy-tls\") pod \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") "
Jul 07 15:22:27.278976 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.278778    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kserve-provision-location\") pod \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") "
Jul 07 15:22:27.278976 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.278814    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kube-rbac-proxy-sar-config\") pod \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") "
Jul 07 15:22:27.278976 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.278839    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-cabundle-cert\") pod \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\" (UID: \"69a1ebc8-7d5e-4f82-9591-e20585bff1cf\") "
Jul 07 15:22:27.279189 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.279166    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69a1ebc8-7d5e-4f82-9591-e20585bff1cf" (UID: "69a1ebc8-7d5e-4f82-9591-e20585bff1cf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:22:27.279250 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.279199    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69a1ebc8-7d5e-4f82-9591-e20585bff1cf" (UID: "69a1ebc8-7d5e-4f82-9591-e20585bff1cf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:22:27.279287 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.279243    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69a1ebc8-7d5e-4f82-9591-e20585bff1cf" (UID: "69a1ebc8-7d5e-4f82-9591-e20585bff1cf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:22:27.280894 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.280873    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kube-api-access-ttrk8" (OuterVolumeSpecName: "kube-api-access-ttrk8") pod "69a1ebc8-7d5e-4f82-9591-e20585bff1cf" (UID: "69a1ebc8-7d5e-4f82-9591-e20585bff1cf"). InnerVolumeSpecName "kube-api-access-ttrk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:22:27.280966 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.280877    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69a1ebc8-7d5e-4f82-9591-e20585bff1cf" (UID: "69a1ebc8-7d5e-4f82-9591-e20585bff1cf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:22:27.380484 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.380449    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kserve-provision-location\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:22:27.380484 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.380479    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:22:27.380484 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.380491    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-cabundle-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:22:27.380737 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.380500    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttrk8\" (UniqueName: \"kubernetes.io/projected/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-kube-api-access-ttrk8\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:22:27.380737 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.380510    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69a1ebc8-7d5e-4f82-9591-e20585bff1cf-proxy-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:22:27.557209 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.557110    2581 generic.go:356] "Generic (PLEG): container finished" podID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerID="8531040f2fbad7184c0f7b2ffa6acd3430cf7f4e7e34456fade970bef92ebd79" exitCode=0
Jul 07 15:22:27.557209 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.557179    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"
Jul 07 15:22:27.557418 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.557198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" event={"ID":"69a1ebc8-7d5e-4f82-9591-e20585bff1cf","Type":"ContainerDied","Data":"8531040f2fbad7184c0f7b2ffa6acd3430cf7f4e7e34456fade970bef92ebd79"}
Jul 07 15:22:27.557418 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.557290    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922" event={"ID":"69a1ebc8-7d5e-4f82-9591-e20585bff1cf","Type":"ContainerDied","Data":"ece10dbd668176e9886f497fc8b738e7e3c62c904b462fa4292d5a054aa82f8d"}
Jul 07 15:22:27.557418 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.557320    2581 scope.go:117] "RemoveContainer" containerID="14c897d5662a1f52f533ec5df307bc981e6139dca7759f912af2b84780eb65d8"
Jul 07 15:22:27.566237 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.566219    2581 scope.go:117] "RemoveContainer" containerID="8531040f2fbad7184c0f7b2ffa6acd3430cf7f4e7e34456fade970bef92ebd79"
Jul 07 15:22:27.573396 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.573378    2581 scope.go:117] "RemoveContainer" containerID="1a1ecdb35f3a88ce82cc1219347c79a6643d83c733b0128bbe7c9decb4b69298"
Jul 07 15:22:27.580577 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.580560    2581 scope.go:117] "RemoveContainer" containerID="14c897d5662a1f52f533ec5df307bc981e6139dca7759f912af2b84780eb65d8"
Jul 07 15:22:27.580806 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:22:27.580785    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c897d5662a1f52f533ec5df307bc981e6139dca7759f912af2b84780eb65d8\": container with ID starting with 14c897d5662a1f52f533ec5df307bc981e6139dca7759f912af2b84780eb65d8 not found: ID does not exist" containerID="14c897d5662a1f52f533ec5df307bc981e6139dca7759f912af2b84780eb65d8"
Jul 07 15:22:27.580860 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.580815    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c897d5662a1f52f533ec5df307bc981e6139dca7759f912af2b84780eb65d8"} err="failed to get container status \"14c897d5662a1f52f533ec5df307bc981e6139dca7759f912af2b84780eb65d8\": rpc error: code = NotFound desc = could not find container \"14c897d5662a1f52f533ec5df307bc981e6139dca7759f912af2b84780eb65d8\": container with ID starting with 14c897d5662a1f52f533ec5df307bc981e6139dca7759f912af2b84780eb65d8 not found: ID does not exist"
Jul 07 15:22:27.580860 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.580832    2581 scope.go:117] "RemoveContainer" containerID="8531040f2fbad7184c0f7b2ffa6acd3430cf7f4e7e34456fade970bef92ebd79"
Jul 07 15:22:27.581083 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:22:27.581062    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8531040f2fbad7184c0f7b2ffa6acd3430cf7f4e7e34456fade970bef92ebd79\": container with ID starting with 8531040f2fbad7184c0f7b2ffa6acd3430cf7f4e7e34456fade970bef92ebd79 not found: ID does not exist" containerID="8531040f2fbad7184c0f7b2ffa6acd3430cf7f4e7e34456fade970bef92ebd79"
Jul 07 15:22:27.581142 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.581088    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8531040f2fbad7184c0f7b2ffa6acd3430cf7f4e7e34456fade970bef92ebd79"} err="failed to get container status \"8531040f2fbad7184c0f7b2ffa6acd3430cf7f4e7e34456fade970bef92ebd79\": rpc error: code = NotFound desc = could not find container \"8531040f2fbad7184c0f7b2ffa6acd3430cf7f4e7e34456fade970bef92ebd79\": container with ID starting with 8531040f2fbad7184c0f7b2ffa6acd3430cf7f4e7e34456fade970bef92ebd79 not found: ID does not exist"
Jul 07 15:22:27.581142 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.581106    2581 scope.go:117] "RemoveContainer" containerID="1a1ecdb35f3a88ce82cc1219347c79a6643d83c733b0128bbe7c9decb4b69298"
Jul 07 15:22:27.581414 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:22:27.581340    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1ecdb35f3a88ce82cc1219347c79a6643d83c733b0128bbe7c9decb4b69298\": container with ID starting with 1a1ecdb35f3a88ce82cc1219347c79a6643d83c733b0128bbe7c9decb4b69298 not found: ID does not exist" containerID="1a1ecdb35f3a88ce82cc1219347c79a6643d83c733b0128bbe7c9decb4b69298"
Jul 07 15:22:27.581501 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.581424    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1ecdb35f3a88ce82cc1219347c79a6643d83c733b0128bbe7c9decb4b69298"} err="failed to get container status \"1a1ecdb35f3a88ce82cc1219347c79a6643d83c733b0128bbe7c9decb4b69298\": rpc error: code = NotFound desc = could not find container \"1a1ecdb35f3a88ce82cc1219347c79a6643d83c733b0128bbe7c9decb4b69298\": container with ID starting with 1a1ecdb35f3a88ce82cc1219347c79a6643d83c733b0128bbe7c9decb4b69298 not found: ID does not exist"
Jul 07 15:22:27.587642 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.587620    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"]
Jul 07 15:22:27.596045 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:27.596025    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-00f07-predictor-5977594885-pc922"]
Jul 07 15:22:28.932335 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:28.932301    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" path="/var/lib/kubelet/pods/69a1ebc8-7d5e-4f82-9591-e20585bff1cf/volumes"
Jul 07 15:22:28.973385 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:28.973336    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sz62b/must-gather-crvpd"]
Jul 07 15:22:28.973714 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:28.973701    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kserve-container"
Jul 07 15:22:28.973762 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:28.973716    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kserve-container"
Jul 07 15:22:28.973762 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:28.973731    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="storage-initializer"
Jul 07 15:22:28.973762 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:28.973736    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="storage-initializer"
Jul 07 15:22:28.973762 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:28.973753    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kube-rbac-proxy"
Jul 07 15:22:28.973762 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:28.973759    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kube-rbac-proxy"
Jul 07 15:22:28.973915 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:28.973814    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kserve-container"
Jul 07 15:22:28.973915 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:28.973821    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a1ebc8-7d5e-4f82-9591-e20585bff1cf" containerName="kube-rbac-proxy"
Jul 07 15:22:28.978490 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:28.978469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sz62b/must-gather-crvpd"
Jul 07 15:22:28.981831 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:28.981810    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sz62b\"/\"default-dockercfg-zh4r2\""
Jul 07 15:22:28.981963 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:28.981859    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sz62b\"/\"kube-root-ca.crt\""
Jul 07 15:22:28.981963 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:28.981874    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sz62b\"/\"openshift-service-ca.crt\""
Jul 07 15:22:28.987328 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:28.987306    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sz62b/must-gather-crvpd"]
Jul 07 15:22:29.097328 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:29.097295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/512f4fe8-33e2-4fb3-9ef6-090ec780d845-must-gather-output\") pod \"must-gather-crvpd\" (UID: \"512f4fe8-33e2-4fb3-9ef6-090ec780d845\") " pod="openshift-must-gather-sz62b/must-gather-crvpd"
Jul 07 15:22:29.097499 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:29.097464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlt6f\" (UniqueName: \"kubernetes.io/projected/512f4fe8-33e2-4fb3-9ef6-090ec780d845-kube-api-access-rlt6f\") pod \"must-gather-crvpd\" (UID: \"512f4fe8-33e2-4fb3-9ef6-090ec780d845\") " pod="openshift-must-gather-sz62b/must-gather-crvpd"
Jul 07 15:22:29.198262 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:29.198164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/512f4fe8-33e2-4fb3-9ef6-090ec780d845-must-gather-output\") pod \"must-gather-crvpd\" (UID: \"512f4fe8-33e2-4fb3-9ef6-090ec780d845\") " pod="openshift-must-gather-sz62b/must-gather-crvpd"
Jul 07 15:22:29.198262 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:29.198230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlt6f\" (UniqueName: \"kubernetes.io/projected/512f4fe8-33e2-4fb3-9ef6-090ec780d845-kube-api-access-rlt6f\") pod \"must-gather-crvpd\" (UID: \"512f4fe8-33e2-4fb3-9ef6-090ec780d845\") " pod="openshift-must-gather-sz62b/must-gather-crvpd"
Jul 07 15:22:29.198553 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:29.198538    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/512f4fe8-33e2-4fb3-9ef6-090ec780d845-must-gather-output\") pod \"must-gather-crvpd\" (UID: \"512f4fe8-33e2-4fb3-9ef6-090ec780d845\") " pod="openshift-must-gather-sz62b/must-gather-crvpd"
Jul 07 15:22:29.209024 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:29.208996    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlt6f\" (UniqueName: \"kubernetes.io/projected/512f4fe8-33e2-4fb3-9ef6-090ec780d845-kube-api-access-rlt6f\") pod \"must-gather-crvpd\" (UID: \"512f4fe8-33e2-4fb3-9ef6-090ec780d845\") " pod="openshift-must-gather-sz62b/must-gather-crvpd"
Jul 07 15:22:29.300498 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:29.300460    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sz62b/must-gather-crvpd"
Jul 07 15:22:29.434825 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:29.434801    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sz62b/must-gather-crvpd"]
Jul 07 15:22:29.436952 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:22:29.436926    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod512f4fe8_33e2_4fb3_9ef6_090ec780d845.slice/crio-0e4f22940881a9b81944ba09f837df06c22c5f7400b1090e0aa96f25d5be5705 WatchSource:0}: Error finding container 0e4f22940881a9b81944ba09f837df06c22c5f7400b1090e0aa96f25d5be5705: Status 404 returned error can't find the container with id 0e4f22940881a9b81944ba09f837df06c22c5f7400b1090e0aa96f25d5be5705
Jul 07 15:22:29.438755 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:29.438740    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 15:22:29.568459 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:29.568371    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sz62b/must-gather-crvpd" event={"ID":"512f4fe8-33e2-4fb3-9ef6-090ec780d845","Type":"ContainerStarted","Data":"0e4f22940881a9b81944ba09f837df06c22c5f7400b1090e0aa96f25d5be5705"}
Jul 07 15:22:36.602591 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:36.602547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sz62b/must-gather-crvpd" event={"ID":"512f4fe8-33e2-4fb3-9ef6-090ec780d845","Type":"ContainerStarted","Data":"2ca64a34ab70773b1d906190e3728a130c3cb51eca017b58c9e4aede33707eaa"}
Jul 07 15:22:36.603173 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:36.602604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sz62b/must-gather-crvpd" event={"ID":"512f4fe8-33e2-4fb3-9ef6-090ec780d845","Type":"ContainerStarted","Data":"36567d7cb3e84c726e6d637dcd3f344d969e65f96d01e2b2fc07b99c06446397"}
Jul 07 15:22:36.624829 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:36.624762    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sz62b/must-gather-crvpd" podStartSLOduration=2.466670844 podStartE2EDuration="8.624741158s" totalImagesPullingTime="6.158070314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:22:36.623887773 +0000 UTC m=+1518.356500092" watchObservedRunningTime="2026-07-07 15:22:36.624741158 +0000 UTC m=+1518.357353482"
Jul 07 15:22:57.700537 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:57.700500    2581 generic.go:356] "Generic (PLEG): container finished" podID="512f4fe8-33e2-4fb3-9ef6-090ec780d845" containerID="36567d7cb3e84c726e6d637dcd3f344d969e65f96d01e2b2fc07b99c06446397" exitCode=0
Jul 07 15:22:57.701072 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:57.700561    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sz62b/must-gather-crvpd" event={"ID":"512f4fe8-33e2-4fb3-9ef6-090ec780d845","Type":"ContainerDied","Data":"36567d7cb3e84c726e6d637dcd3f344d969e65f96d01e2b2fc07b99c06446397"}
Jul 07 15:22:57.701072 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:57.700965    2581 scope.go:117] "RemoveContainer" containerID="36567d7cb3e84c726e6d637dcd3f344d969e65f96d01e2b2fc07b99c06446397"
Jul 07 15:22:58.089414 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:58.089332    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sz62b_must-gather-crvpd_512f4fe8-33e2-4fb3-9ef6-090ec780d845/gather/0.log"
Jul 07 15:22:58.652656 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:58.652615    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tv5km/must-gather-lhrzc"]
Jul 07 15:22:58.655206 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:58.655187    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tv5km/must-gather-lhrzc"
Jul 07 15:22:58.658184 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:58.658160    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tv5km\"/\"default-dockercfg-lljgq\""
Jul 07 15:22:58.658312 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:58.658280    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tv5km\"/\"openshift-service-ca.crt\""
Jul 07 15:22:58.658378 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:58.658331    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tv5km\"/\"kube-root-ca.crt\""
Jul 07 15:22:58.669670 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:58.669646    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tv5km/must-gather-lhrzc"]
Jul 07 15:22:58.756112 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:58.756072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mk4h\" (UniqueName: \"kubernetes.io/projected/40db945c-2279-42a6-b1b0-ae34389ff17e-kube-api-access-9mk4h\") pod \"must-gather-lhrzc\" (UID: \"40db945c-2279-42a6-b1b0-ae34389ff17e\") " pod="openshift-must-gather-tv5km/must-gather-lhrzc"
Jul 07 15:22:58.756573 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:58.756127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/40db945c-2279-42a6-b1b0-ae34389ff17e-must-gather-output\") pod \"must-gather-lhrzc\" (UID: \"40db945c-2279-42a6-b1b0-ae34389ff17e\") " pod="openshift-must-gather-tv5km/must-gather-lhrzc"
Jul 07 15:22:58.857192 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:58.857143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mk4h\" (UniqueName: \"kubernetes.io/projected/40db945c-2279-42a6-b1b0-ae34389ff17e-kube-api-access-9mk4h\") pod \"must-gather-lhrzc\" (UID: \"40db945c-2279-42a6-b1b0-ae34389ff17e\") " pod="openshift-must-gather-tv5km/must-gather-lhrzc"
Jul 07 15:22:58.857436 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:58.857209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/40db945c-2279-42a6-b1b0-ae34389ff17e-must-gather-output\") pod \"must-gather-lhrzc\" (UID: \"40db945c-2279-42a6-b1b0-ae34389ff17e\") " pod="openshift-must-gather-tv5km/must-gather-lhrzc"
Jul 07 15:22:58.857618 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:58.857595    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/40db945c-2279-42a6-b1b0-ae34389ff17e-must-gather-output\") pod \"must-gather-lhrzc\" (UID: \"40db945c-2279-42a6-b1b0-ae34389ff17e\") " pod="openshift-must-gather-tv5km/must-gather-lhrzc"
Jul 07 15:22:58.866757 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:58.866733    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mk4h\" (UniqueName: \"kubernetes.io/projected/40db945c-2279-42a6-b1b0-ae34389ff17e-kube-api-access-9mk4h\") pod \"must-gather-lhrzc\" (UID: \"40db945c-2279-42a6-b1b0-ae34389ff17e\") " pod="openshift-must-gather-tv5km/must-gather-lhrzc"
Jul 07 15:22:58.964553 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:58.964516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tv5km/must-gather-lhrzc"
Jul 07 15:22:59.099387 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:59.097521    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tv5km/must-gather-lhrzc"]
Jul 07 15:22:59.100728 ip-10-0-143-168 kubenswrapper[2581]: W0707 15:22:59.100696    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40db945c_2279_42a6_b1b0_ae34389ff17e.slice/crio-0a3e3407ad9e3ac2abc936dfd38bcfd64a0d7fa1ac0829e8d590348832a46ae2 WatchSource:0}: Error finding container 0a3e3407ad9e3ac2abc936dfd38bcfd64a0d7fa1ac0829e8d590348832a46ae2: Status 404 returned error can't find the container with id 0a3e3407ad9e3ac2abc936dfd38bcfd64a0d7fa1ac0829e8d590348832a46ae2
Jul 07 15:22:59.710502 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:22:59.710462    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tv5km/must-gather-lhrzc" event={"ID":"40db945c-2279-42a6-b1b0-ae34389ff17e","Type":"ContainerStarted","Data":"0a3e3407ad9e3ac2abc936dfd38bcfd64a0d7fa1ac0829e8d590348832a46ae2"}
Jul 07 15:23:00.718105 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:00.718058    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tv5km/must-gather-lhrzc" event={"ID":"40db945c-2279-42a6-b1b0-ae34389ff17e","Type":"ContainerStarted","Data":"40f7bb36662ee35da86f4fcff7671fc9395df36def27b67e124e112ffebc77d3"}
Jul 07 15:23:00.718572 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:00.718120    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tv5km/must-gather-lhrzc" event={"ID":"40db945c-2279-42a6-b1b0-ae34389ff17e","Type":"ContainerStarted","Data":"a3e60d6d7652b9382e686a85b92174413a6cef6a56f41397ca212020db344120"}
Jul 07 15:23:00.752048 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:00.751982    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tv5km/must-gather-lhrzc" podStartSLOduration=2.100673082 podStartE2EDuration="2.75195738s" totalImagesPullingTime="651.284298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:23:00.749881003 +0000 UTC m=+1542.482493346" watchObservedRunningTime="2026-07-07 15:23:00.75195738 +0000 UTC m=+1542.484569701"
Jul 07 15:23:01.258098 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:01.258064    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b9d6j_386c6191-61a2-412c-9331-152a3d94baf9/global-pull-secret-syncer/0.log"
Jul 07 15:23:01.514404 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:01.514304    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kwvxp_2b9620a4-5364-4242-9b08-e05f0989e587/konnectivity-agent/0.log"
Jul 07 15:23:01.592337 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:01.592303    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-168.ec2.internal_648d8ccbe686a4d38db4c29e47b7d68c/haproxy/0.log"
Jul 07 15:23:03.648023 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:03.647974    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-sz62b/must-gather-crvpd"]
Jul 07 15:23:03.648699 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:03.648296    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-sz62b/must-gather-crvpd" podUID="512f4fe8-33e2-4fb3-9ef6-090ec780d845" containerName="copy" containerID="cri-o://2ca64a34ab70773b1d906190e3728a130c3cb51eca017b58c9e4aede33707eaa" gracePeriod=2
Jul 07 15:23:03.652936 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:03.652616    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-sz62b/must-gather-crvpd"]
Jul 07 15:23:03.652936 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:23:03.652901    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-crvpd\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sz62b\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="512f4fe8-33e2-4fb3-9ef6-090ec780d845" pod="openshift-must-gather-sz62b/must-gather-crvpd"
Jul 07 15:23:04.022645 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.022528    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sz62b_must-gather-crvpd_512f4fe8-33e2-4fb3-9ef6-090ec780d845/copy/0.log"
Jul 07 15:23:04.028258 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.028226    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sz62b/must-gather-crvpd"
Jul 07 15:23:04.031038 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:23:04.030974    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-crvpd\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sz62b\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="512f4fe8-33e2-4fb3-9ef6-090ec780d845" pod="openshift-must-gather-sz62b/must-gather-crvpd"
Jul 07 15:23:04.107302 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.106564    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/512f4fe8-33e2-4fb3-9ef6-090ec780d845-must-gather-output\") pod \"512f4fe8-33e2-4fb3-9ef6-090ec780d845\" (UID: \"512f4fe8-33e2-4fb3-9ef6-090ec780d845\") "
Jul 07 15:23:04.107302 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.106724    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlt6f\" (UniqueName: \"kubernetes.io/projected/512f4fe8-33e2-4fb3-9ef6-090ec780d845-kube-api-access-rlt6f\") pod \"512f4fe8-33e2-4fb3-9ef6-090ec780d845\" (UID: \"512f4fe8-33e2-4fb3-9ef6-090ec780d845\") "
Jul 07 15:23:04.108994 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.108926    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512f4fe8-33e2-4fb3-9ef6-090ec780d845-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "512f4fe8-33e2-4fb3-9ef6-090ec780d845" (UID: "512f4fe8-33e2-4fb3-9ef6-090ec780d845"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:23:04.112829 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.112786    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512f4fe8-33e2-4fb3-9ef6-090ec780d845-kube-api-access-rlt6f" (OuterVolumeSpecName: "kube-api-access-rlt6f") pod "512f4fe8-33e2-4fb3-9ef6-090ec780d845" (UID: "512f4fe8-33e2-4fb3-9ef6-090ec780d845"). InnerVolumeSpecName "kube-api-access-rlt6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:23:04.207934 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.207856    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlt6f\" (UniqueName: \"kubernetes.io/projected/512f4fe8-33e2-4fb3-9ef6-090ec780d845-kube-api-access-rlt6f\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:23:04.207934 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.207901    2581 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/512f4fe8-33e2-4fb3-9ef6-090ec780d845-must-gather-output\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 07 15:23:04.748225 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.748196    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sz62b_must-gather-crvpd_512f4fe8-33e2-4fb3-9ef6-090ec780d845/copy/0.log"
Jul 07 15:23:04.749629 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.749601    2581 generic.go:356] "Generic (PLEG): container finished" podID="512f4fe8-33e2-4fb3-9ef6-090ec780d845" containerID="2ca64a34ab70773b1d906190e3728a130c3cb51eca017b58c9e4aede33707eaa" exitCode=143
Jul 07 15:23:04.749875 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.749862    2581 scope.go:117] "RemoveContainer" containerID="2ca64a34ab70773b1d906190e3728a130c3cb51eca017b58c9e4aede33707eaa"
Jul 07 15:23:04.750158 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.750143    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sz62b/must-gather-crvpd"
Jul 07 15:23:04.762978 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:23:04.762944    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-crvpd\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sz62b\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="512f4fe8-33e2-4fb3-9ef6-090ec780d845" pod="openshift-must-gather-sz62b/must-gather-crvpd"
Jul 07 15:23:04.777554 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:23:04.775220    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-crvpd\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sz62b\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="512f4fe8-33e2-4fb3-9ef6-090ec780d845" pod="openshift-must-gather-sz62b/must-gather-crvpd"
Jul 07 15:23:04.777554 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.775476    2581 scope.go:117] "RemoveContainer" containerID="36567d7cb3e84c726e6d637dcd3f344d969e65f96d01e2b2fc07b99c06446397"
Jul 07 15:23:04.801593 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.801040    2581 scope.go:117] "RemoveContainer" containerID="2ca64a34ab70773b1d906190e3728a130c3cb51eca017b58c9e4aede33707eaa"
Jul 07 15:23:04.802813 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:23:04.802777    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca64a34ab70773b1d906190e3728a130c3cb51eca017b58c9e4aede33707eaa\": container with ID starting with 2ca64a34ab70773b1d906190e3728a130c3cb51eca017b58c9e4aede33707eaa not found: ID does not exist" containerID="2ca64a34ab70773b1d906190e3728a130c3cb51eca017b58c9e4aede33707eaa"
Jul 07 15:23:04.802947 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.802838    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca64a34ab70773b1d906190e3728a130c3cb51eca017b58c9e4aede33707eaa"} err="failed to get container status \"2ca64a34ab70773b1d906190e3728a130c3cb51eca017b58c9e4aede33707eaa\": rpc error: code = NotFound desc = could not find container \"2ca64a34ab70773b1d906190e3728a130c3cb51eca017b58c9e4aede33707eaa\": container with ID starting with 2ca64a34ab70773b1d906190e3728a130c3cb51eca017b58c9e4aede33707eaa not found: ID does not exist"
Jul 07 15:23:04.802947 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.802867    2581 scope.go:117] "RemoveContainer" containerID="36567d7cb3e84c726e6d637dcd3f344d969e65f96d01e2b2fc07b99c06446397"
Jul 07 15:23:04.804673 ip-10-0-143-168 kubenswrapper[2581]: E0707 15:23:04.804644    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36567d7cb3e84c726e6d637dcd3f344d969e65f96d01e2b2fc07b99c06446397\": container with ID starting with 36567d7cb3e84c726e6d637dcd3f344d969e65f96d01e2b2fc07b99c06446397 not found: ID does not exist" containerID="36567d7cb3e84c726e6d637dcd3f344d969e65f96d01e2b2fc07b99c06446397"
Jul 07 15:23:04.804808 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.804681    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36567d7cb3e84c726e6d637dcd3f344d969e65f96d01e2b2fc07b99c06446397"} err="failed to get container status \"36567d7cb3e84c726e6d637dcd3f344d969e65f96d01e2b2fc07b99c06446397\": rpc error: code = NotFound desc = could not find container \"36567d7cb3e84c726e6d637dcd3f344d969e65f96d01e2b2fc07b99c06446397\": container with ID starting with 36567d7cb3e84c726e6d637dcd3f344d969e65f96d01e2b2fc07b99c06446397 not found: ID does not exist"
Jul 07 15:23:04.937086 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:04.937044    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="512f4fe8-33e2-4fb3-9ef6-090ec780d845" path="/var/lib/kubelet/pods/512f4fe8-33e2-4fb3-9ef6-090ec780d845/volumes"
Jul 07 15:23:05.969913 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:05.969874    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_329e599d-b769-4855-940e-2dbb2f95bf2b/alertmanager/0.log"
Jul 07 15:23:05.997588 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:05.997553    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_329e599d-b769-4855-940e-2dbb2f95bf2b/config-reloader/0.log"
Jul 07 15:23:06.030267 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.030233    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_329e599d-b769-4855-940e-2dbb2f95bf2b/kube-rbac-proxy-web/0.log"
Jul 07 15:23:06.055416 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.055370    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_329e599d-b769-4855-940e-2dbb2f95bf2b/kube-rbac-proxy/0.log"
Jul 07 15:23:06.081471 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.081441    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_329e599d-b769-4855-940e-2dbb2f95bf2b/kube-rbac-proxy-metric/0.log"
Jul 07 15:23:06.105181 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.105154    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_329e599d-b769-4855-940e-2dbb2f95bf2b/prom-label-proxy/0.log"
Jul 07 15:23:06.130121 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.130057    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_329e599d-b769-4855-940e-2dbb2f95bf2b/init-config-reloader/0.log"
Jul 07 15:23:06.211594 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.211505    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-5lx8n_2e6d5ccf-86b8-49d2-9a73-835584742a65/kube-state-metrics/0.log"
Jul 07 15:23:06.236338 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.236263    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-5lx8n_2e6d5ccf-86b8-49d2-9a73-835584742a65/kube-rbac-proxy-main/0.log"
Jul 07 15:23:06.260786 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.260759    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-5lx8n_2e6d5ccf-86b8-49d2-9a73-835584742a65/kube-rbac-proxy-self/0.log"
Jul 07 15:23:06.300663 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.300550    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-898cf4bf8-hnjwl_fbc65dc2-5780-48ac-82a6-47e65b143112/metrics-server/0.log"
Jul 07 15:23:06.329807 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.329778    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-zq9ns_c4cf86be-6802-4c24-92fd-49f31354feaf/monitoring-plugin/0.log"
Jul 07 15:23:06.507522 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.507416    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tq54m_334b7bba-a8fa-4589-98db-4f59626c7da6/node-exporter/0.log"
Jul 07 15:23:06.528961 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.528932    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tq54m_334b7bba-a8fa-4589-98db-4f59626c7da6/kube-rbac-proxy/0.log"
Jul 07 15:23:06.552006 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.551977    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tq54m_334b7bba-a8fa-4589-98db-4f59626c7da6/init-textfile/0.log"
Jul 07 15:23:06.579073 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.579048    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-xbxlx_faad4c48-420e-4d7d-9673-1d63790e2c33/kube-rbac-proxy-main/0.log"
Jul 07 15:23:06.600134 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.600105    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-xbxlx_faad4c48-420e-4d7d-9673-1d63790e2c33/kube-rbac-proxy-self/0.log"
Jul 07 15:23:06.625165 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.625127    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-xbxlx_faad4c48-420e-4d7d-9673-1d63790e2c33/openshift-state-metrics/0.log"
Jul 07 15:23:06.666872 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.666843    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6704654e-8d88-4f83-bac2-c55b696da1f6/prometheus/0.log"
Jul 07 15:23:06.686163 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.686133    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6704654e-8d88-4f83-bac2-c55b696da1f6/config-reloader/0.log"
Jul 07 15:23:06.715900 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.715866    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6704654e-8d88-4f83-bac2-c55b696da1f6/thanos-sidecar/0.log"
Jul 07 15:23:06.746601 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.746572    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6704654e-8d88-4f83-bac2-c55b696da1f6/kube-rbac-proxy-web/0.log"
Jul 07 15:23:06.773669 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.773590    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6704654e-8d88-4f83-bac2-c55b696da1f6/kube-rbac-proxy/0.log"
Jul 07 15:23:06.799685 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.799630    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6704654e-8d88-4f83-bac2-c55b696da1f6/kube-rbac-proxy-thanos/0.log"
Jul 07 15:23:06.824330 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.824279    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6704654e-8d88-4f83-bac2-c55b696da1f6/init-config-reloader/0.log"
Jul 07 15:23:06.862870 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.862831    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-jdh4f_754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6/prometheus-operator/0.log"
Jul 07 15:23:06.884743 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.884719    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-jdh4f_754b8f0b-4a44-4d63-aa46-2d1cc4d6a6f6/kube-rbac-proxy/0.log"
Jul 07 15:23:06.917077 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.917025    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-bzfwg_0277897e-a284-4a9c-84a1-3e05e767f3e6/prometheus-operator-admission-webhook/0.log"
Jul 07 15:23:06.952711 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.952678    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55856b8684-c4p2j_10799fc0-1122-4432-b46e-13894dd6d7db/telemeter-client/0.log"
Jul 07 15:23:06.986408 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:06.986376    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55856b8684-c4p2j_10799fc0-1122-4432-b46e-13894dd6d7db/reload/0.log"
Jul 07 15:23:07.013369 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:07.013298    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55856b8684-c4p2j_10799fc0-1122-4432-b46e-13894dd6d7db/kube-rbac-proxy/0.log"
Jul 07 15:23:07.052361 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:07.052241    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6f9bc8bb-lspfk_504ebd6e-ecc5-4a6a-9afc-59648f933fb4/thanos-query/0.log"
Jul 07 15:23:07.105332 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:07.104831    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6f9bc8bb-lspfk_504ebd6e-ecc5-4a6a-9afc-59648f933fb4/kube-rbac-proxy-web/0.log"
Jul 07 15:23:07.162037 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:07.161957    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6f9bc8bb-lspfk_504ebd6e-ecc5-4a6a-9afc-59648f933fb4/kube-rbac-proxy/0.log"
Jul 07 15:23:07.205255 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:07.205231    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6f9bc8bb-lspfk_504ebd6e-ecc5-4a6a-9afc-59648f933fb4/prom-label-proxy/0.log"
Jul 07 15:23:07.240015 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:07.239976    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6f9bc8bb-lspfk_504ebd6e-ecc5-4a6a-9afc-59648f933fb4/kube-rbac-proxy-rules/0.log"
Jul 07 15:23:07.271563 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:07.271529    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6f9bc8bb-lspfk_504ebd6e-ecc5-4a6a-9afc-59648f933fb4/kube-rbac-proxy-metrics/0.log"
Jul 07 15:23:09.729828 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.729765    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"]
Jul 07 15:23:09.730331 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.730236    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="512f4fe8-33e2-4fb3-9ef6-090ec780d845" containerName="gather"
Jul 07 15:23:09.730331 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.730251    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="512f4fe8-33e2-4fb3-9ef6-090ec780d845" containerName="gather"
Jul 07 15:23:09.730331 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.730264    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="512f4fe8-33e2-4fb3-9ef6-090ec780d845" containerName="copy"
Jul 07 15:23:09.730331 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.730270    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="512f4fe8-33e2-4fb3-9ef6-090ec780d845" containerName="copy"
Jul 07 15:23:09.730331 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.730323    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="512f4fe8-33e2-4fb3-9ef6-090ec780d845" containerName="copy"
Jul 07 15:23:09.730331 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.730332    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="512f4fe8-33e2-4fb3-9ef6-090ec780d845" containerName="gather"
Jul 07 15:23:09.735605 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.735580    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:09.762960 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.762923    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"]
Jul 07 15:23:09.863895 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.863852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/531d9cf4-995c-46bc-a709-05ae15e892ec-lib-modules\") pod \"perf-node-gather-daemonset-w68vv\" (UID: \"531d9cf4-995c-46bc-a709-05ae15e892ec\") " pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:09.864087 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.863967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/531d9cf4-995c-46bc-a709-05ae15e892ec-proc\") pod \"perf-node-gather-daemonset-w68vv\" (UID: \"531d9cf4-995c-46bc-a709-05ae15e892ec\") " pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:09.864087 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.864029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/531d9cf4-995c-46bc-a709-05ae15e892ec-podres\") pod \"perf-node-gather-daemonset-w68vv\" (UID: \"531d9cf4-995c-46bc-a709-05ae15e892ec\") " pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:09.864193 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.864090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/531d9cf4-995c-46bc-a709-05ae15e892ec-sys\") pod \"perf-node-gather-daemonset-w68vv\" (UID: \"531d9cf4-995c-46bc-a709-05ae15e892ec\") " pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:09.864193 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.864159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddfg4\" (UniqueName: \"kubernetes.io/projected/531d9cf4-995c-46bc-a709-05ae15e892ec-kube-api-access-ddfg4\") pod \"perf-node-gather-daemonset-w68vv\" (UID: \"531d9cf4-995c-46bc-a709-05ae15e892ec\") " pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:09.965263 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.965217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddfg4\" (UniqueName: \"kubernetes.io/projected/531d9cf4-995c-46bc-a709-05ae15e892ec-kube-api-access-ddfg4\") pod \"perf-node-gather-daemonset-w68vv\" (UID: \"531d9cf4-995c-46bc-a709-05ae15e892ec\") " pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:09.965602 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.965575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/531d9cf4-995c-46bc-a709-05ae15e892ec-lib-modules\") pod \"perf-node-gather-daemonset-w68vv\" (UID: \"531d9cf4-995c-46bc-a709-05ae15e892ec\") " pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:09.965779 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.965761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/531d9cf4-995c-46bc-a709-05ae15e892ec-proc\") pod \"perf-node-gather-daemonset-w68vv\" (UID: \"531d9cf4-995c-46bc-a709-05ae15e892ec\") " pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:09.965918 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.965902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/531d9cf4-995c-46bc-a709-05ae15e892ec-podres\") pod \"perf-node-gather-daemonset-w68vv\" (UID: \"531d9cf4-995c-46bc-a709-05ae15e892ec\") " pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:09.966076 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.966040    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/531d9cf4-995c-46bc-a709-05ae15e892ec-podres\") pod \"perf-node-gather-daemonset-w68vv\" (UID: \"531d9cf4-995c-46bc-a709-05ae15e892ec\") " pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:09.966146 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.965800    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/531d9cf4-995c-46bc-a709-05ae15e892ec-lib-modules\") pod \"perf-node-gather-daemonset-w68vv\" (UID: \"531d9cf4-995c-46bc-a709-05ae15e892ec\") " pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:09.966146 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.966058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/531d9cf4-995c-46bc-a709-05ae15e892ec-sys\") pod \"perf-node-gather-daemonset-w68vv\" (UID: \"531d9cf4-995c-46bc-a709-05ae15e892ec\") " pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:09.966253 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.965850    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/531d9cf4-995c-46bc-a709-05ae15e892ec-proc\") pod \"perf-node-gather-daemonset-w68vv\" (UID: \"531d9cf4-995c-46bc-a709-05ae15e892ec\") " pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:09.966406 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.966387    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/531d9cf4-995c-46bc-a709-05ae15e892ec-sys\") pod \"perf-node-gather-daemonset-w68vv\" (UID: \"531d9cf4-995c-46bc-a709-05ae15e892ec\") " pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:09.974085 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:09.974056    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddfg4\" (UniqueName: \"kubernetes.io/projected/531d9cf4-995c-46bc-a709-05ae15e892ec-kube-api-access-ddfg4\") pod \"perf-node-gather-daemonset-w68vv\" (UID: \"531d9cf4-995c-46bc-a709-05ae15e892ec\") " pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:10.046789 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:10.046704    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:10.230875 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:10.230750    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"]
Jul 07 15:23:10.801794 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:10.801757    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv" event={"ID":"531d9cf4-995c-46bc-a709-05ae15e892ec","Type":"ContainerStarted","Data":"b7040ef36211f6eb73ceede89d5d00f7ed0affbda4790a2d98207fc7cb62cbf2"}
Jul 07 15:23:10.802289 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:10.801808    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv" event={"ID":"531d9cf4-995c-46bc-a709-05ae15e892ec","Type":"ContainerStarted","Data":"af3dc53069653a7ca4b2372bf6bfbb797374dd5f0a8e9bca06f1d325a06405fd"}
Jul 07 15:23:10.802289 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:10.801855    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:10.826217 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:10.826162    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv" podStartSLOduration=1.82614485 podStartE2EDuration="1.82614485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:23:10.82293478 +0000 UTC m=+1552.555547112" watchObservedRunningTime="2026-07-07 15:23:10.82614485 +0000 UTC m=+1552.558757170"
Jul 07 15:23:11.059862 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:11.059785    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gdcl2_5521ca14-0a45-4824-bf15-fac6fce0e2f0/dns/0.log"
Jul 07 15:23:11.083162 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:11.083136    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gdcl2_5521ca14-0a45-4824-bf15-fac6fce0e2f0/kube-rbac-proxy/0.log"
Jul 07 15:23:11.216380 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:11.216323    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7fg5g_4c0798f1-c292-4322-bc4a-79dda10ef0c9/dns-node-resolver/0.log"
Jul 07 15:23:11.775029 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:11.775001    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-72b7k_13881bc5-ab4d-4258-9e74-195665decbf9/node-ca/0.log"
Jul 07 15:23:13.003680 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:13.003651    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lmjpm_3bf7e0cf-669e-42c4-8ee8-76764abc256a/serve-healthcheck-canary/0.log"
Jul 07 15:23:13.460754 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:13.460727    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fm8rl_2a34eac8-0d11-49d8-bd6a-9bfec15e3144/kube-rbac-proxy/0.log"
Jul 07 15:23:13.480287 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:13.480264    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fm8rl_2a34eac8-0d11-49d8-bd6a-9bfec15e3144/exporter/0.log"
Jul 07 15:23:13.500903 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:13.500873    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fm8rl_2a34eac8-0d11-49d8-bd6a-9bfec15e3144/extractor/0.log"
Jul 07 15:23:16.132058 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:16.132016    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-b8dcz_ccc1e2f1-8be8-4f60-ac53-5ca78eda47d2/manager/0.log"
Jul 07 15:23:16.164393 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:16.164363    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-k7q4c_2c44693e-46aa-48c6-b326-f57b52e70db1/s3-init/0.log"
Jul 07 15:23:16.182323 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:16.182292    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-k7q4c_2c44693e-46aa-48c6-b326-f57b52e70db1/download-hf-model/0.log"
Jul 07 15:23:16.817570 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:16.817535    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tv5km/perf-node-gather-daemonset-w68vv"
Jul 07 15:23:22.319148 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:22.319118    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wwsc4_ce04ed69-ff4c-438a-9261-9f3c66fc6c09/kube-multus-additional-cni-plugins/0.log"
Jul 07 15:23:22.340767 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:22.340730    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wwsc4_ce04ed69-ff4c-438a-9261-9f3c66fc6c09/egress-router-binary-copy/0.log"
Jul 07 15:23:22.365314 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:22.365288    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wwsc4_ce04ed69-ff4c-438a-9261-9f3c66fc6c09/cni-plugins/0.log"
Jul 07 15:23:22.386136 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:22.386109    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wwsc4_ce04ed69-ff4c-438a-9261-9f3c66fc6c09/bond-cni-plugin/0.log"
Jul 07 15:23:22.407304 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:22.407280    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wwsc4_ce04ed69-ff4c-438a-9261-9f3c66fc6c09/routeoverride-cni/0.log"
Jul 07 15:23:22.427701 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:22.427680    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wwsc4_ce04ed69-ff4c-438a-9261-9f3c66fc6c09/whereabouts-cni-bincopy/0.log"
Jul 07 15:23:22.447615 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:22.447590    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wwsc4_ce04ed69-ff4c-438a-9261-9f3c66fc6c09/whereabouts-cni/0.log"
Jul 07 15:23:22.511385 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:22.511342    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n292g_b63fcac7-de40-44b0-bdce-81d2e93c8403/kube-multus/0.log"
Jul 07 15:23:22.564646 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:22.564613    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bjq65_2c77f4a4-6fbf-4ca3-8291-a48491c2f423/network-metrics-daemon/0.log"
Jul 07 15:23:22.583459 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:22.583385    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bjq65_2c77f4a4-6fbf-4ca3-8291-a48491c2f423/kube-rbac-proxy/0.log"
Jul 07 15:23:23.688644 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:23.688616    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q5z8m_ee1eaa63-f9de-41bc-8127-3d5082be4471/ovn-controller/0.log"
Jul 07 15:23:23.759235 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:23.759209    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q5z8m_ee1eaa63-f9de-41bc-8127-3d5082be4471/ovn-acl-logging/0.log"
Jul 07 15:23:23.813543 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:23.813518    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q5z8m_ee1eaa63-f9de-41bc-8127-3d5082be4471/kube-rbac-proxy-node/0.log"
Jul 07 15:23:23.847705 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:23.847661    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q5z8m_ee1eaa63-f9de-41bc-8127-3d5082be4471/kube-rbac-proxy-ovn-metrics/0.log"
Jul 07 15:23:23.871213 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:23.871184    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q5z8m_ee1eaa63-f9de-41bc-8127-3d5082be4471/northd/0.log"
Jul 07 15:23:23.896470 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:23.896448    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q5z8m_ee1eaa63-f9de-41bc-8127-3d5082be4471/nbdb/0.log"
Jul 07 15:23:23.923990 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:23.923960    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q5z8m_ee1eaa63-f9de-41bc-8127-3d5082be4471/sbdb/0.log"
Jul 07 15:23:23.977913 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:23.977820    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q5z8m_ee1eaa63-f9de-41bc-8127-3d5082be4471/ovnkube-controller/0.log"
Jul 07 15:23:25.396366 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:25.396322    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qk5j7_0043d8b0-5817-4b30-950b-0145e8690f3f/network-check-target-container/0.log"
Jul 07 15:23:26.483758 ip-10-0-143-168 kubenswrapper[2581]: I0707 15:23:26.483674    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-kbdbz_213b89a8-eaa5-45bb-a245-20d21e692c38/iptables-alerter/0.log"